Services
What keeps a web server running when nobody is logged in?
A service is a program that systemd starts at boot, restarts on failure and logs; systemctl controls it.
You can start a web server by typing its name in a terminal. Why does nobody run one that way? Because then it starts only when someone types it; a service starts at boot. If it crashes, it stays down; a service is started again. When you log out, it ends with your session; a service does not care who is logged in. And whatever it printed is gone with the terminal, where a service’s output is kept. The program that does all this is called systemd.
Starting and enabling
start is for now; enable is for every boot after
systemctl start runs a service now; systemctl enable links it into its target for boot. enable --now does both.
This is the distinction that catches people out. Start runs the service now. Enable makes it start at every boot, by creating a link from its target. Stop and disable undo each half. So a service you started but never enabled vanishes after the next reboot. Use enable dash dash now to do both at once. And mask goes further: a masked unit cannot be started at all.
Service status
Enabled, running, and what it last said, in one screen
systemctl status shows whether a unit is enabled, its state, its main PID and cgroup, and its last log lines.
When a service misbehaves, start with systemctl status. The first line names it. Loaded tells you where its file is and whether it is enabled. Active says whether it is running, and since when; a failed service shows the exit code here. Main PID is the process systemd watches, and the cgroup lists every process it started. Underneath are its most recent log lines.
The journal
One indexed log for everything, filtered in an instant
journald collects kernel, service and syslog messages; journalctl filters by unit, boot, priority and time.
Logs no longer live in a dozen separate files. The kernel, every service’s output, and every program that writes to syslog feed one journal, which is indexed. So you ask questions: messages from one unit, since this boot, errors and worse, in the last hour, or follow live. The filters combine. One catch: on some systems the journal lives in memory and is lost at reboot unless the directory var log journal exists.
Service units
A short text file turns a program into a managed service
A service unit names the program, its user, its order, its Restart policy and the target it belongs to.
Here is a complete service. A description. After says start once the network and the database are up; Wants pulls the network target in. ExecStart is the program, which stays in the foreground. User means it does not run as root. Restart on failure brings it back five seconds after a crash. And WantedBy says which target it joins when you enable it. No PID files, no log files: systemd watches the process and captures its output.
Service units · Going deeper · 1 of 2
Everything systemd runs is a unit, and /etc always wins
A unit file describes a service, socket, timer, target, mount or path; files in /etc/systemd/system override packaged ones.
systemd calls everything it manages a unit, and each unit is a short text file. Services run programs. Sockets and paths start services on demand. Timers start them on a schedule. Targets group units into a goal, like a fully booted server. Packages put their units under user lib. Yours go in et-see systemd system, and anything there overrides the packaged copy. To change one setting safely, use systemctl edit.
Service units · Going deeper · 2 of 2
Wants pulls a unit in; After only decides the order
Wants and Requires start other units; After orders them. Without After, systemd starts things in parallel.
At boot, systemd works from the default target outward. Multi-user target wants the SSH server, our orders service and cron, so all three are started. But wanting is not ordering. Without After, systemd would start orders and its database at the same moment. After makes orders wait until postgres is up. The same goes for the network. So remember: Wants and Requires say what, After says when.
Boot process
Firmware, bootloader, kernel, initramfs, then systemd
Boot passes from UEFI to GRUB to the kernel and initramfs, and systemd starts everything else.
Starting a Linux machine is a relay race. The firmware checks the hardware and finds a disk to boot from. The bootloader, GRUB, loads the kernel and a small starter file system called the initramfs. The kernel brings up the CPUs, memory and drivers. The initramfs finds the real root disk, even if it is encrypted or on LVM, and mounts it. Then systemd takes over as process one and starts every service. systemd-analyze shows how long each leg took.
GRUB · Going deeper · 1 of 2
Change boot options with the tools, or once at the menu
Edit /etc/default/grub and regenerate, or use grubby on RHEL; press e at the menu for a one-off change.
Boot process · Going deeper · 2 of 2
A target is a state of the whole system; the default is where boot ends
Servers boot to multi-user.target; rescue.target and emergency.target are for repairs. set-default and isolate switch them.
Targets replace the old numbered runlevels. Multi-user is the normal state of a server: every service running and a text login. Graphical adds a desktop. Rescue gives you a root shell with your disks mounted and little else. Emergency trusts only the root file system, mounted read-only. get-default and set-default control where boot ends, and isolate switches the running system now.
Scheduled jobs
Timers, cron lines and at jobs: three ways to run later
A systemd timer starts a service on a schedule; a cron line has five time fields and a command; at runs once.
Three tools run things later. A systemd timer is a unit that starts a service. This one, from Ubuntu itself, fires at six and eighteen hundred every day, adds a random delay so a whole fleet does not hit the mirrors at once, and catches up after the machine was off. cron does the same with one line: minute, hour, day of month, month, day of week, then the command. And at runs a command once, at a given time.
Time synchronisation · Going deeper
chrony keeps the clock right, and everything depends on it
chronyd synchronises with time servers listed in /etc/chrony.conf; timedatectl and chronyc sources check it.
Lab
Write a service that survives a crash and a reboot
A unit file, enable --now, a kill -9 and an automatic restart, a drop-in, the journal and three kinds of schedule.
In this lab you write your own service, enable it, kill it and watch systemd bring it back, override a setting, read its journal, schedule a job three ways and prove it all survives a reboot. Press the down arrow for the steps.
Lab · Going deeper
Your own service, restarted, logged and scheduled
Use a VM where you are root. The unit runs a shell loop, so no software needs installing.
Use a VM where you are root. The unit runs a shell loop, so no software needs installing. Steps. Write the unit. /etc/systemd/system/hello.service with Description=Hello loop, ExecStart=/bin/bash -c 'while true; do echo hello; sleep 10; done', Restart=on-failure, RestartSec=5, and WantedBy=multi-user.target. Load and start it. systemctl daemon-reload; systemctl enable --now hello; systemctl status hello. Find the symlink enable created. Crash it. kill -9 the main PID and watch systemctl status and journalctl -u hello -f. Then systemctl stop hello and confirm it stays stopped. Override a setting. systemctl edit hello, set RestartSec=1 in a drop-in, and confirm with systemctl cat hello. Change targets. systemctl get-default; systemctl isolate rescue.target from a console (not SSH), then reboot back to the default. Read the journal. journalctl -b -p err; journalctl --since "10 min ago" -u hello. Make it persistent: mkdir -p /var/log/journal; systemctl restart systemd-journald. Schedule it three ways. A timer: hello-once.timer with OnCalendar=*:0/5. A cron line in crontab -e. An at job: echo "logger from at" | at now + 2 minutes. Check list-timers, crontab -l and atq. Check the time. timedatectl; chronyc sources. Add a server line to /etc/chrony.conf, restart chronyd and see it in the sources. Survive a reboot. Reboot. hello must be active, the timer listed, and the boot visible in journalctl --list-boots. Check: After a reboot, systemctl is-enabled hello says enabled, is-active says active, and journalctl -u hello -b shows it restarting after your kill -9 before the reboot (with persistent storage).
Steps
- Write the unit. /etc/systemd/system/hello.service with Description=Hello loop, ExecStart=/bin/bash -c 'while true; do echo hello; sleep 10; done', Restart=on-failure, RestartSec=5, and WantedBy=multi-user.target.
- Load and start it. systemctl daemon-reload; systemctl enable --now hello; systemctl status hello. Find the symlink enable created.
- Crash it. kill -9 the main PID and watch systemctl status and journalctl -u hello -f. Then systemctl stop hello and confirm it stays stopped.
- Override a setting. systemctl edit hello, set RestartSec=1 in a drop-in, and confirm with systemctl cat hello.
- Change targets. systemctl get-default; systemctl isolate rescue.target from a console (not SSH), then reboot back to the default.
- Read the journal. journalctl -b -p err; journalctl --since "10 min ago" -u hello. Make it persistent: mkdir -p /var/log/journal; systemctl restart systemd-journald.
- Schedule it three ways. A timer: hello-once.timer with OnCalendar=*:0/5. A cron line in crontab -e. An at job: echo "logger from at" | at now + 2 minutes. Check list-timers, crontab -l and atq.
- Check the time. timedatectl; chronyc sources. Add a server line to /etc/chrony.conf, restart chronyd and see it in the sources.
- Survive a reboot. Reboot. hello must be active, the timer listed, and the boot visible in journalctl --list-boots.
Lesson complete
You finished “Services and start-up”
Nicely done.