Six questions
A server cannot be reached. What could be wrong?
An address, a route, name resolution, saved settings, a listening socket and the firewall: one command checks each.
Someone says the new server cannot be reached. What could be wrong? Only six things, and each has a command. Does it have an address at all? Does it know where to send packets for other networks? Can it turn names into addresses? Are those settings saved, or will they vanish at the next reboot? Is the service actually listening? And is the firewall letting the traffic in? The pages of this lesson take them one at a time.
Interfaces
Each network card has a name, a MAC address and IP addresses
ip -br addr lists each interface, its state and its addresses; names such as enp1s0 come from where the card sits.
Start by asking the machine what it has. ip dash br addr gives one line per interface. lo is loopback, the machine talking to itself. enp1s0 is the Ethernet card: en for Ethernet, p1 s0 for its place on the bus. It has an IPv4 address, a global IPv6 address and a link-local one. ip link shows that it is up, with a link, and its hardware address.
Bonds and bridges · Going deeper
Two cards behind one address, or a switch inside the host
A bond joins cards for failover or speed; a bridge puts virtual machines on the real network.
Subnets and routes
Local addresses are reached directly; the rest go to the gateway
The prefix splits network from host; the routing table sends local traffic directly and the rest to the default gateway.
The slash twenty-four means the first twenty-four bits name the network, leaving eight bits for two hundred and fifty-four hosts. ip route shows the table: one route for the local subnet, and a default route through the gateway. A packet for a machine on the same subnet goes straight to it. A packet for anywhere else goes to the gateway, which passes it on.
Routing and NAT · Going deeper
A Linux host as a router, with port forwarding and masquerade
With IP forwarding on, a host routes; masquerade hides a private network and a port forward exposes one inside service.
Name resolution
The hosts file is read before DNS, and getent shows what programs see
nsswitch.conf sets the order: /etc/hosts first, then the DNS server named in resolv.conf; getent follows it, dig asks DNS only.
When a program asks for a name, the request goes through the system library, which reads nsswitch to learn the order: files, then DNS. The hosts file is checked first, and if the name is there, the search stops. Otherwise resolv.conf names the DNS server, which answers. That is why getent and dig can disagree: getent follows the same path as every program, dig asks DNS only. On Ubuntu, a local cache called systemd-resolved sits in between.
NetworkManager
Changes made with ip vanish; profiles saved with nmcli stay
ip changes the running kernel only; nmcli saves a connection profile that is applied at every boot.
There are two ways to set an address. ip changes the running kernel: instant, and gone at the next reboot. nmcli changes a saved NetworkManager profile, with address, gateway, DNS and IPv6, then applies it; it comes back at every boot. Devices are the cards; connections are the saved profiles. Ubuntu Server writes the same settings in netplan YAML.
Sockets
ss shows who is listening, and on which address
ss -tlnp lists listening TCP ports with their programs; 0.0.0.0 means every address, 127.0.0.1 this machine only.
ss dash t l n p lists every listening TCP port and the program behind it. The flags: TCP, listening, numbers, and processes. sshd and nginx listen on 0.0.0.0, every address, so other machines can reach them if the firewall agrees. postgres listens on 127.0.0.1, this machine only; no firewall rule can change that. The square brackets are the IPv6 equivalent.
Reverse proxy · Going deeper
One front door that shares requests among several servers
A reverse proxy such as nginx ends TLS on one address and balances requests across backend servers.
firewalld
Zones list what may come in, and only saved rules survive
firewalld gives each interface a zone of allowed services and ports; make changes --permanent, then reload.
On RHEL, firewalld puts the network card in a zone, here the public zone, and allows in only what that zone lists. Add a service with dash dash permanent, which saves it but does not apply it yet. Reload, and the saved rules go live. The zone now allows cockpit, DHCP for IPv6, http and ssh, and port 8443. Anything else that arrives is rejected.
firewalld · Going deeper · 1 of 2
A packet passes five checkpoints, and the firewall guards input
Packets meet prerouting, a routing decision, then input or forward; local traffic leaves through output and postrouting.
Follow a packet through the host. It arrives and meets prerouting, where port forwarding can rewrite its destination. The kernel decides: is this for me, or passing through? If it is for this host, it passes the input checkpoint, where a server firewall decides what may reach programs such as sshd. If not, it goes through forward, only when forwarding is on, and postrouting, where masquerade can rewrite the source. Replies and new connections from local programs leave through output and postrouting.
nftables · Going deeper · 2 of 2
The rules underneath: tables, chains, a policy and sets
nftables attaches chains of rules to hooks; the first matching accept or drop wins and the policy decides the rest.
Troubleshooting
Test from the bottom up and stop at the first failure
Check link, address, route, name, listening port and firewall in that order; each depends on the one before.
When something cannot be reached, climb the ladder from the bottom. Is the link up? Is the address right? Is there a route, and does the gateway answer? Does the name resolve to the right address? Is the service listening, on the right address? Can the client reach the port through the firewall? The first check that fails is where the fault is.
Lab
A fixed address, a name and an open port
A static IPv4 and IPv6 address, a hostname, a hosts entry, a web service through the firewall, a static route and a port forward.
In this lab you set up a server the way you would in production: a fixed address and name, a local hosts entry, a web service opened in the firewall, then a static route and a port forward. You break it once on purpose, and reboot to prove it all comes back. Press the down arrow for the steps.
Lab · Going deeper
A fixed address, a name and an open port
Use a RHEL-family VM (AlmaLinux or Rocky are free). Keep its console open: a wrong address can cut your SSH session. Replace the example addresses with ones on your own network.
Use a RHEL-family VM (AlmaLinux or Rocky are free). Keep its console open: a wrong address can cut your SSH session. Replace the example addresses with ones on your own network. Steps. Read the current state. ip -br addr; ip route; nmcli dev; nmcli con show; cat /etc/resolv.conf. Note the connection name, address, gateway and DNS. Make it static. nmcli con mod "<name>" ipv4.method manual ipv4.addresses <addr>/24 ipv4.gateway <gw> ipv4.dns <dns>; nmcli con up "<name>". Reconnect if you were moved. Add IPv6. nmcli con mod "<name>" ipv6.method manual ipv6.addresses 2001:db8:10::21/64; nmcli con up; ip -6 addr. Name it. hostnamectl set-hostname web01.lab; add a line for another machine to /etc/hosts; compare getent hosts with dig for the same name. Publish a service. Install and enable nginx. curl -I http://localhost works; from the second VM, curl -I http://<addr> should time out or be refused. Open the firewall. firewall-cmd --permanent --add-service=http; firewall-cmd --reload; --list-all. The outside curl now works. Bind it locally. Change nginx to listen 127.0.0.1:80, reload, and watch ss -tlnp and the outside curl. Change it back. Route and forward. Add a static route with nmcli con mod "<name>" +ipv4.routes "10.20.0.0/16 <gw>"; ip route get 10.20.4.1. Add a forward port 8080 to 80 with firewall-cmd and test it from outside. Prove it. Reboot. The address, IPv6, hostname, route, firewall rules and nginx must all come back. Check: After a reboot, ip -br addr shows both addresses, hostnamectl shows web01.lab, and curl -I http://<addr> and http://<addr>:8080 from another machine both return HTTP/1.1 200 OK.
Steps
- Read the current state. ip -br addr; ip route; nmcli dev; nmcli con show; cat /etc/resolv.conf. Note the connection name, address, gateway and DNS.
- Make it static. nmcli con mod "<name>" ipv4.method manual ipv4.addresses <addr>/24 ipv4.gateway <gw> ipv4.dns <dns>; nmcli con up "<name>". Reconnect if you were moved.
- Add IPv6. nmcli con mod "<name>" ipv6.method manual ipv6.addresses 2001:db8:10::21/64; nmcli con up; ip -6 addr.
- Name it. hostnamectl set-hostname web01.lab; add a line for another machine to /etc/hosts; compare getent hosts with dig for the same name.
- Publish a service. Install and enable nginx. curl -I http://localhost works; from the second VM, curl -I http://<addr> should time out or be refused.
- Open the firewall. firewall-cmd --permanent --add-service=http; firewall-cmd --reload; --list-all. The outside curl now works.
- Bind it locally. Change nginx to listen 127.0.0.1:80, reload, and watch ss -tlnp and the outside curl. Change it back.
- Route and forward. Add a static route with nmcli con mod "<name>" +ipv4.routes "10.20.0.0/16 <gw>"; ip route get 10.20.4.1. Add a forward port 8080 to 80 with firewall-cmd and test it from outside.
- Prove it. Reboot. The address, IPv6, hostname, route, firewall rules and nginx must all come back.
Lesson complete
You finished “Networking”
Nicely done.