Services

What keeps a web server running when nobody is logged in?

Started by hand in a terminalRun as a serviceStarts✗when someone types it✓at boot, with nobody logged inIf it crashes✗it stays down✓it is started againWhen you log out✗it ends with your session✓it keeps runningWhat it prints✗goes to your terminal, then is lost✓is kept in the journalA service is a program that systemd starts, watches and restarts, so that it is there without you.A web server, a database and SSH itself all run as services; the older word is daemon. systemd is the first processthe kernel starts (PID 1), and systemctl is how you talk to it. Four things cover the daily work, and the next pages takethem in turn: start and stop a service, enable it for every boot, read its status, and read what it logged.

A service is a program that systemd starts at boot, restarts on failure and logs; systemctl controls it.

Starting and enabling

start is for now; enable is for every boot after

nowat every bootturn onturn offsystemctl start nginxruns it now; gone after rebootsystemctl enable nginxlinks it into multi-user.target.wantssystemctl stop nginxstops it now; back after rebootsystemctl disable nginxremoves the linksystemctl enable --now nginx both at oncesystemctl mask nginx cannot be started at all, even by handstart and stop act now; enable and disable decide the next boot. You almost always want both.enable only creates a symbolic link from the unit’s WantedBy target to the unit file. The classic exam mistake is a servicethat works, then is gone after the reboot: it was started but never enabled. systemctl is-enabled and is-active check each half.

systemctl start runs a service now; systemctl enable links it into its target for boot. enable --now does both.

Service status

Enabled, running, and what it last said, in one screen

$ systemctl status orders● orders.service - Orders API1 Loaded: loaded (/etc/systemd/system/orders.service; enabled; ...)2 Active: active (running) since Sun 2026-11-15 09:12:04 UTC; 2h ago3 Main PID: 1824 (server)4 Tasks: 9 (limit: 19007) Memory: 61.2M (peak: 74.9M) CPU: 1min 12.402s CGroup: /system.slice/orders.service └─1824 /opt/orders/bin/server --port 8080Nov 15 09:12:04 web01 systemd[1]: Started orders.service - Orders API.51The unit and its description● is green when running2Where its file is; enabled?enabled: starts at boot3Running, and since whenfailed would be red, with a code4The process systemd watchesand every child, in its cgroup5Its last log linesthe full story: journalctl -usystemctl status answers the three questions first: is it enabled, is it running, and what did it last say.A crashed service shows Active: failed with the exit code and the last ten log lines, which is usually enough to find the cause.systemctl --failed lists every failed unit on the machine; systemctl reset-failed clears them once fixed.

systemctl status shows whether a unit is enabled, its state, its main PID and cgroup, and its last log lines.

The journal

One indexed log for everything, filtered in an instant

kernelservices: stdout and stderrsyslog() callssystemd-journaldindexed, binaryQueryShowsjournalctl -u nginxone unit’s messagesjournalctl -bsince this boot; -b -1 the boot beforejournalctl -p errpriority error and worsejournalctl --since "1 hour ago"a time windowjournalctl -ffollow new lines, like tail -fjournalctl -u nginx -b -p warningfilters combineKept across reboots only if /var/log/journal exists (Storage=persistent in journald.conf).rsyslog can still write classic text files under /var/log.Every message from the kernel and every service goes into one indexed journal, queried by unit, boot, time and priority.Because it is indexed, filters are instant even on large logs, and every line carries the unit, PID and boot it came from.Priorities run from 0 emerg to 7 debug. journalctl --disk-usage and --vacuum-size=500M keep its size in check.

journald collects kernel, service and syslog messages; journalctl filters by unit, boot, priority and time.

Service units

A short text file turns a program into a managed service

/etc/systemd/system/orders.service[Unit]Description=Orders API1After=network-online.target postgresql.service2Wants=network-online.target[Service]Type=simpleUser=orders4ExecStart=/opt/orders/bin/server --port 80803Restart=on-failure5RestartSec=5[Install]WantedBy=multi-user.target61What it isshown by status and in the journal2When to start, and what to pull inAfter: wait for them; Wants: start them3The program, in the foregroundType=simple: it must not fork away4Not as rootthe account the process runs as5Bring it back if it crashesnot when stopped on purpose6Start at boot, once enabledjoins multi-user.targetA service unit says what to run, as whom, after what, what to do when it dies, and when it belongs.After a change: systemctl daemon-reload, then restart. systemd keeps the process in the foreground, catches its output for the journaland restarts it according to Restart=, so the program needs no daemon code, PID file or log file of its own.

A service unit names the program, its user, its order, its Restart policy and the target it belongs to.

Service units · Going deeper · 1 of 2

Everything systemd runs is a unit, and /etc always wins

Unit typeDescribesExample.servicea program to run and keep runningnginx.service.socketa port that starts a service on demandcups.socket.timera schedule that starts a serviceapt-daily.timer.targeta group of units: a boot goalmulti-user.target.mounta file system to mounthome.mount.patha path that starts a service when it changescups.pathWhere unit files live/etc/systemd/system/yours: wins over all▲ overrides/run/systemd/system/made at runtime▲ overrides/usr/lib/systemd/system/from packages: do not editsystemd manages units, each described by a small text file, and the copy in /etc always wins.Packages install units under /usr/lib; your own units and changes go in /etc/systemd/system. To change one setting of a packagedunit, run systemctl edit nginx: it makes a drop-in file that overrides just that setting and survives package updates.systemctl cat nginx shows the unit with every drop-in applied; systemctl list-units --type=service lists what is loaded.

A unit file describes a service, socket, timer, target, mount or path; files in /etc/systemd/system override packaged ones.

Service units · Going deeper · 2 of 2

Wants pulls a unit in; After only decides the order

multi-user.targetsshd.serviceorders.servicecrond.serviceWantspostgresql.serviceAfter: start orders only once this is upnetwork-online.targetWants pull in, start anyway if it failsRequires pull in, stop if it stopsAfter order only: start laterWants and Requires say what else to start; After says only in what order. A unit usually needs both.Without After, systemd starts everything at once, which is why it boots fast; orders could start before its database is ready.systemctl list-dependencies multi-user.target shows the tree.

Wants and Requires start other units; After orders them. Without After, systemd starts things in parallel.

Boot process

Firmware, bootloader, kernel, initramfs, then systemd

firmware2.1 sUEFI checks hardware, finds the diskbootloader1 sGRUB loads kernel + initramfskernel1.4 sstarts CPUs, memory, driversinitramfs2.9 sfinds and mounts the real rootsystemd (userspace)6.2 sstarts every service in order13.6 s to multi-user.target$ systemd-analyze$ systemd-analyze blame | headwhich units took longestBoot is a relay: firmware, bootloader, kernel, initramfs, then systemd starts everything else.Each stage only has to find and start the next. The initramfs is a small file system in memory that holds the drivers needed to reachthe real root disk, which might be on LVM, RAID or encrypted. Most of the time, and most of the problems, are in the last stage:systemd-analyze shows how long each took. Times are one virtual machine’s, to scale.

Boot passes from UEFI to GRUB to the kernel and initramfs, and systemd starts everything else.

GRUB · Going deeper · 1 of 2

Change boot options with the tools, or once at the menu

/etc/default/grubGRUB_TIMEOUT, GRUB_CMDLINE_LINUXgenerate/boot/grub2/grub.cfgnever edit by handUbuntu: sudo update-grubRHEL: grub2-mkconfig -o /boot/grub2/grub.cfgRHEL, one change for every kernel:# grubby --update-kernel=ALL --args="quiet"# grubby --info=DEFAULTAt the menu, for one boot onlypress e, find the line starting linux,add to its end:systemd.unit=rescue.targetsystemd.unit=emergency.targetrd.breakthen Ctrl-X to boot.Lesson 14 uses this to reseta lost root password.Change boot options in /etc/default/grub or with grubby, never in the generated grub.cfg.The kernel command line is how you pass options such as a target to boot into or a console. Permanent changes go through the tools,so a kernel update keeps them; a one-off change is made at the menu and forgotten at the next boot. On UEFI systems RHEL keepsgrub.cfg under /boot/grub2/ and a small loader in /boot/efi/EFI/redhat/.

Edit /etc/default/grub and regenerate, or use grubby on RHEL; press e at the menu for a one-off change.

Boot process · Going deeper · 2 of 2

A target is a state of the whole system; the default is where boot ends

Targetold runlevelMeanspoweroff.target0offrescue.target1single user: root shell, local disks mountedmulti-user.target3every service, text login: the server defaultgraphical.target5multi-user plus a desktopreboot.target6restartemergency.targetroot shell, root mounted read-only, nothing else$ systemctl get-defaultmulti-user.target# systemctl set-default graphical.target# systemctl isolate rescue.targetA target is a named state of the whole system, and the default target is where boot ends.set-default changes it for every boot; isolate switches now, stopping anything the new target does not need. Rescue andemergency are for repairs: rescue mounts your disks and starts very little, emergency trusts nothing but the root file system.

Servers boot to multi-user.target; rescue.target and emergency.target are for repairs. set-default and isolate switch them.

Scheduled jobs

Timers, cron lines and at jobs: three ways to run later

/usr/lib/systemd/system/apt-daily.timer[Timer]OnCalendar=*-*-* 6,18:001RandomizedDelaySec=12h2Persistent=true3[Install]WantedBy=timers.target1every day at 06:00 and 18:002plus a random delay, so a fleet does not all start at once3missed while off? run it at the next bootcron: one line per job (crontab -e)0minute2hour*day of month*month1-5day of week (0 = Sun)cmd0 2 * * 1-5 /usr/local/bin/backup.sh02:00, Monday to Fridayat: once, later$ echo "systemctl reboot" | at 03:00$ atq; atrm 3A timer is a unit that starts a service on a schedule; cron does the same with one line, and at runs a job once.Timers log to the journal, can catch up after downtime and appear in systemctl list-timers with their next run time. cron is simplerand still everywhere. systemd-analyze calendar "Mon..Fri 02:00" checks an OnCalendar expression before you rely on it.

A systemd timer starts a service on a schedule; a cron line has five time fields and a command; at runs once.

Time synchronisation · Going deeper

chrony keeps the clock right, and everything depends on it

$ timedatectl Local time: Sun 2026-11-15 10:42:17 UTC Time zone: Etc/UTC (UTC, +0000)System clock synchronized: yes NTP service: active$ chronyc sourcesMS Name/IP address Stratum Poll Reach LastRx Last sample^* time.cloudflare.com 3 10 377 412 +118us^+ ntp1.example.net 2 10 377 398 -402us1Is the clock synchronised?timedatectl answers in one line2^* is the source in use^+ is a good backup3Servers in /etc/chrony.confserver NAME iburst4Then restart chronydand check chronyc sourceschrony keeps the clock right by asking time servers; every log, certificate and Kerberos ticket depends on it.Set the zone with timedatectl set-timezone Europe/London. Add or change servers in /etc/chrony.conf (pool lines on Ubuntu),restart chronyd (chrony on Ubuntu) and confirm a ^* source. The Red Hat exam asks you to point a client at a given server.

chronyd synchronises with time servers listed in /etc/chrony.conf; timedatectl and chronyc sources check it.

Lab

Write a service that survives a crash and a reboot

# vim /etc/systemd/system/hello.service1# systemctl daemon-reload; systemctl enable --now hello2Created symlink /etc/systemd/system/multi-user.target.wants/hello.service → /etc/systemd/system/hello.service.# kill -9 $(systemctl show -p MainPID --value hello)3# journalctl -u hello -n 3 --no-pager4Nov 15 10:51:03 lab systemd[1]: hello.service: Main process exited, code=killed, status=9/KILLNov 15 10:51:08 lab systemd[1]: hello.service: Scheduled restart job, restart counter is at 1.Nov 15 10:51:08 lab systemd[1]: Started hello.service - Hello loop.# systemctl list-timers --all | head -351Write a unita loop that logs every 10 s2Load, enable, startenable made the link3Kill it hardRestart=on-failure4Watch systemd bring it backfive seconds later5Then schedule itwith a timerLab: write a service, make it survive a crash and a reboot, then put it on a timer.Needs a VM with systemd and root. Press the down arrow for the steps.

A unit file, enable --now, a kill -9 and an automatic restart, a drop-in, the journal and three kinds of schedule.

Lab · Going deeper

Your own service, restarted, logged and scheduled

Use a VM where you are root. The unit runs a shell loop, so no software needs installing.

Steps

  1. Write the unit. /etc/systemd/system/hello.service with Description=Hello loop, ExecStart=/bin/bash -c 'while true; do echo hello; sleep 10; done', Restart=on-failure, RestartSec=5, and WantedBy=multi-user.target.
  2. Load and start it. systemctl daemon-reload; systemctl enable --now hello; systemctl status hello. Find the symlink enable created.
  3. Crash it. kill -9 the main PID and watch systemctl status and journalctl -u hello -f. Then systemctl stop hello and confirm it stays stopped.
  4. Override a setting. systemctl edit hello, set RestartSec=1 in a drop-in, and confirm with systemctl cat hello.
  5. Change targets. systemctl get-default; systemctl isolate rescue.target from a console (not SSH), then reboot back to the default.
  6. Read the journal. journalctl -b -p err; journalctl --since "10 min ago" -u hello. Make it persistent: mkdir -p /var/log/journal; systemctl restart systemd-journald.
  7. Schedule it three ways. A timer: hello-once.timer with OnCalendar=*:0/5. A cron line in crontab -e. An at job: echo "logger from at" | at now + 2 minutes. Check list-timers, crontab -l and atq.
  8. Check the time. timedatectl; chronyc sources. Add a server line to /etc/chrony.conf, restart chronyd and see it in the sources.
  9. Survive a reboot. Reboot. hello must be active, the timer listed, and the boot visible in journalctl --list-boots.
Check: After a reboot, systemctl is-enabled hello says enabled, is-active says active, and journalctl -u hello -b shows it restarting after your kill -9 before the reboot (with persistent storage).

Lesson complete

You finished “Services and start-up”

Nicely done.

Open in a new tab