Six questions

A server cannot be reached. What could be wrong?

QuestionCommand that answers itPage in this lesson1Does this server have an address?ip addrInterfaces2Where do packets for other networks leave?ip routeSubnets and routes3Do names turn into addresses?getent hosts nameName resolution4Will the settings survive a reboot?nmcliNetworkManager5Is the service listening, and on which address?ss -tlnpSockets6Does the firewall let the traffic in?firewall-cmd --list-allfirewalldSix questions, asked in this order, find most network faults on a server. Each has one command.Addresses, subnets, gateways and DNS themselves are taught in the Networking course. This lesson is about where eachof them lives on a Linux server, how to read it, and how to set it so that it is still set after a reboot.The last page of the lesson walks the same list as a troubleshooting method.

An address, a route, name resolution, saved settings, a listening socket and the firewall: one command checks each.

Interfaces

Each network card has a name, a MAC address and IP addresses

$ ip -br addrlo UNKNOWN 127.0.0.1/8 ::1/1281enp1s0 UP 192.168.10.21/24 2001:db8:10::21/64 fe80::5054:ff:fe3a:1c2e/642$ ip link show enp1s02: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default3 link/ether 52:54:00:3a:1c:2e brd ff:ff:ff:ff:ff:ff41lo, loopbackthe machine talking to itself2enp1s0: en Ethernet, p1 PCI bus 1, s0 slot 0IPv4 with its /24, a global IPv6, and an fe80:: link-localaddress that every IPv6 interface has3UP: switched onLOWER_UP: a cable and a link4the MAC address of the cardEach interface has a name, a hardware (MAC) address, a link state and one or more IP addresses.Names are predictable: they come from where the card sits (enp1s0, ens3, eno1), so they survive reboots and new hardware.ip -br addr is the quick view; ip addr, ip link and ip -6 addr give detail. ifconfig and route are retired: use ip.Example output from a RHEL 10 virtual machine; addresses are from the ranges reserved for documentation.

ip -br addr lists each interface, its state and its addresses; names such as enp1s0 come from where the card sits.

Bonds and bridges · Going deeper

Two cards behind one address, or a switch inside the host

bond: two cards, one addresseno1eno2bond0192.168.10.21active-backup: one card works, the other waits802.3ad (LACP): both work; the switch must agreebridge: a switch inside the hostenp1s0br0192.168.10.21vnet0vnet1VMs join the real network with their own addresses# nmcli con add type bond ifname bond0 bond.options "mode=active-backup,miimon=100"# nmcli con add type ethernet ifname eno1 master bond0; nmcli con add type ethernet ifname eno2 master bond0# nmcli con add type bridge ifname br0; nmcli con add type bridge-slave ifname enp1s0 master br0A bond joins cards for failover or speed; a bridge lets virtual machines share the host’s network.The address moves to bond0 or br0; the physical cards become its ports and carry no address of their own. cat /proc/net/bonding/bond0shows which card is active. Keep a console open when you move the address off the card you are connected through.

A bond joins cards for failover or speed; a bridge puts virtual machines on the real network.

Subnets and routes

Local addresses are reached directly; the rest go to the gateway

192.168.10.21/24: 32 bits, the first 24 name the network192.168.10.21network part: 24 bitshost part: 8 bits = 254 hostsnetwork 192.168.10.0broadcast 192.168.10.255$ ip routedefault via 192.168.10.1 dev enp1s0 proto static metric 1001192.168.10.0/24 dev enp1s0 proto kernel scope link src 192.168.10.21 metric 10022To 192.168.10.40: on my own subnet, so straight to it, by its MAC address.1To 8.8.8.8: no closer route matches, so to the default gateway 192.168.10.1, which passes it on.A host delivers to its own subnet directly and sends everything else to its default gateway.The prefix splits an address: /24 leaves 8 host bits, 256 values, 254 usable once the network and broadcast addresses are taken.For each packet the kernel picks the most specific route that matches; default (0.0.0.0/0) matches anything and is used last.IPv6 works the same way with longer addresses: a /64 is the normal subnet size. ip -6 route shows its table.

The prefix splits network from host; the routing table sends local traffic directly and the rest to the default gateway.

Routing and NAT · Going deeper

A Linux host as a router, with port forwarding and masquerade

private network 10.0.0.0/24app 10.0.0.1010.0.0.25gw01: a Linux routereth1 10.0.0.1eth0 203.0.113.10net.ipv4.ip_forward = 1internetin to 203.0.113.10:8080 → rewritten to 10.0.0.10:80 (DNAT, a port forward)out from 10.0.0.25 → leaves as 203.0.113.10 (masquerade)# echo net.ipv4.ip_forward=1 > /etc/sysctl.d/90-forward.conf; sysctl --system# firewall-cmd --permanent --zone=external --add-masquerade# firewall-cmd --permanent --zone=external --add-forward-port=port=8080:proto=tcp:toport=80:toaddr=10.0.0.10# nmcli con mod eth1 +ipv4.routes "10.20.0.0/16 10.0.0.254" a static route to another networkA Linux host becomes a router with one setting, and firewalld adds the address rewriting.With forwarding on, packets that are not for this host go out again. Masquerade gives a private network one public address on theway out; a forward port sends an outside port to an inside machine. A static route says which gateway reaches a network thedefault route does not. Check routes with ip route get 10.20.4.1, which prints the route the kernel would pick.

With IP forwarding on, a host routes; masquerade hides a private network and a port forward exposes one inside service.

Name resolution

The hosts file is read before DNS, and getent shows what programs see

a programasks for db01getaddrinfo()nsswitch.confhosts: files dnsthe order to ask/etc/hosts192.168.10.30 db01found? stop hereresolv.confnameserver ...which DNS serverDNS serverdb01 → 10.0.0.30answers by networkTwo tools, two answers:$ getent hosts db01192.168.10.30 db01the same path as every program: hosts file first$ dig +short db0110.0.0.30DNS only: never reads /etc/hostsOn Ubuntu, resolv.conf says nameserver 127.0.0.53: systemd-resolved, a local cache. resolvectl status shows the real servers.Names are looked up in the order nsswitch.conf gives: usually the hosts file first, then DNS.A stale line in /etc/hosts beats DNS for every program on the machine, while dig, which asks DNS directly, shows the right address.When they disagree, getent shows what programs really get. Set DNS servers through the connection profile (ipv4.dns), not byediting resolv.conf, which NetworkManager or systemd-resolved rewrites.

nsswitch.conf sets the order: /etc/hosts first, then the DNS server named in resolv.conf; getent follows it, dig asks DNS only.

NetworkManager

Changes made with ip vanish; profiles saved with nmcli stay

ip: the running kernel only# ip addr add 192.168.10.50/24 \ dev enp1s0Works at once; gone at the nextreboot or when the profile isre-applied. Good for a quick test.nmcli: a saved profile, applied at every boot# nmcli con mod enp1s0 ipv4.method manual \ ipv4.addresses 192.168.10.21/24 ipv4.gateway 192.168.10.1 \ ipv4.dns 192.168.10.2 ipv6.addresses 2001:db8:10::21/64# nmcli con up enp1s0stored in /etc/NetworkManager/system-connections/enp1s0.nmconnectionDevice or connection?nmcli dev: the cards. nmcli con: saved profiles,one active per device. nmtui gives menus.Ubuntu Server: netplan/etc/netplan/50-cloud-init.yaml then netplan applyYAML that netplan hands to systemd-networkd (servers) or NetworkManager (desktops)Settings made with ip vanish at reboot; settings saved with nmcli or netplan are applied at every boot.On RHEL, NetworkManager owns the network: change a profile with nmcli con mod, then nmcli con up to apply it. The name is set thesame way for good: hostnamectl set-hostname web01.lab. The old ifcfg files are not read on RHEL 10; profiles are keyfiles.

ip changes the running kernel only; nmcli saves a connection profile that is applied at every boot.

Sockets

ss shows who is listening, and on which address

# ss -tlnpState Recv-Q Send-Q Local Address:Port Peer Address:Port ProcessLISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=812,fd=3))1LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1204,fd=6))1LISTEN 0 200 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=990,fd=7))2LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=812,fd=4))3-t TCP -l listening -n numbers, not names -p the program (as root)10.0.0.0: every IPv4 addressreachable from the networkif the firewall allows it2127.0.0.1: this machine onlyno firewall rule can open it;the program must listen wider3[::]: every IPv6 addressss -tn shows live connections;-u for UDP (DNS, chrony)ss shows which programs listen on which ports, and on which address: all of them, or only this machine.A service that works locally but not from outside is usually bound to 127.0.0.1, or blocked by the firewall: ss tells the two apart.From another machine, nc -zv web01 80 or curl -v http://web01 tests the same port end to end. netstat is retired: use ss.

ss -tlnp lists listening TCP ports with their programs; 0.0.0.0 means every address, 127.0.0.1 this machine only.

Reverse proxy · Going deeper

One front door that shares requests among several servers

clientsHTTPS, port 443nginx on web01ends TLS, picks a backendlistens on 0.0.0.0:443in turnapp1:8080app2:8080in the server blockupstream app { server app1:8080; server app2:8080;}location / { proxy_pass http://app;}A reverse proxy takes every request on one address and hands it to one of several backends.Clients see one name and one certificate; the apps behind it can be restarted or added without them noticing. nginx sends requeststo the upstream servers in turn and skips one that stops answering. On RHEL, SELinux blocks nginx from opening connections untilsetsebool -P httpd_can_network_connect on. Test the config with nginx -t before systemctl reload nginx.

A reverse proxy such as nginx ends TLS on one address and balances requests across backend servers.

firewalld

Zones list what may come in, and only saved rules survive

# firewall-cmd --get-active-zonespublic interfaces: enp1s0# firewall-cmd --permanent --add-service=http1success# firewall-cmd --reload2success# firewall-cmd --list-allpublic (default, active) interfaces: enp1s0 services: cockpit dhcpv6-client http ssh3 ports: 8443/tcp1--permanent writes the rule to diskin /etc/firewalld; it is not active yet2--reload makes the saved rules liveand drops runtime-only changes3Allowed in: these services, these ports.Anything else arriving is rejected.Without --permanent: live now, lost at reload.--runtime-to-permanent keeps what works.Ubuntu: ufw allow 80/tcp does the same job.firewalld puts each interface in a zone and allows in only the services and ports that zone lists.A service is a named set of ports (http is TCP 80; see /usr/lib/firewalld/services). Add one with --add-service, or a bare port with--add-port=8443/tcp. Every change is runtime or --permanent: do it --permanent, then --reload, then check with --list-all.

firewalld gives each interface a zone of allowed services and ports; make changes --permanent, then reload.

firewalld · Going deeper · 1 of 2

A packet passes five checkpoints, and the firewall guards input

in from the networkpreroutingDNAT: port forwardingroutingfor me, or passing on?for meinputfilter what may reach messhd, nginxa local programnot for meforwardonly if ip_forward=1postroutingSNAT, masqueradeoutoutputreplies, new connectionsEvery packet passes a fixed set of checkpoints; the firewall rules for this server sit at input.Arriving packets meet prerouting, where port forwarding rewrites the destination, then a routing decision. Packets for this host gothrough input to a program; others go through forward and out again, which a plain server refuses. Replies leave through outputand postrouting, where masquerade rewrites the source. nftables and firewalld are two ways to write rules at these points.

Packets meet prerouting, a routing decision, then input or forward; local traffic leaves through output and postrouting.

nftables · Going deeper · 2 of 2

The rules underneath: tables, chains, a policy and sets

# nft list ruleset (a hand-written ruleset)table inet filter { chain input {1 type filter hook input priority 0; policy drop;2 ct state established,related accept3 iif "lo" accept tcp dport { 22, 80, 443 } accept4 icmp type echo-request accept }}1table inet: IPv4 and IPv6 togethertables hold chains2Hook input, policy dropwhat does not match is dropped3Replies to our own trafficconnection tracking lets them in4Open ports, as a setone rule for three portsnftables is the kernel’s packet filter; firewalld writes nftables rules for you.A chain attaches to a hook (input, forward, output...) and runs its rules in order; the first accept or drop wins, and the policydecides the rest. nft list ruleset shows every rule in force, including firewalld’s. Use one manager at a time: firewalld orhand-written nftables (saved in /etc/sysconfig/nftables.conf on RHEL), not both.

nftables attaches chains of rules to hooks; the first matching accept or drop wins and the policy decides the rest.

Troubleshooting

Test from the bottom up and stop at the first failure

CheckCommandA healthy answer1Linkip linkstate UP, LOWER_UP2Addressip addrthe address and prefix you expect3Routeip route; ping -c3 192.168.10.1a default route; the gateway replies4Namegetent hosts db01the right address5Listeningss -tlnp (on the server)the port, on the right address6Firewall and pathnc -zv db01 5432 (from the client)"succeeded"; if not, firewall-cmd --list-allTest a network fault from the bottom up: link, address, route, name, port, firewall. Stop at the first failure.Each check depends on the ones before: no route means no DNS, and no DNS looks like a dead server. ping can be blocked by afirewall while the service works, so test the real port too. tracepath or mtr shows where along the path packets stop.

Check link, address, route, name, listening port and firewall in that order; each depends on the one before.

Lab

A fixed address, a name and an open port

# nmcli con mod enp1s0 ipv4.method manual \1 ipv4.addresses 192.168.10.21/24 ipv4.gateway 192.168.10.1 \ ipv4.dns 192.168.10.2; nmcli con up enp1s0# hostnamectl set-hostname web01.lab2# echo "192.168.10.30 db01.lab db01" >> /etc/hosts; getent hosts db013192.168.10.30 db01.lab db01# dnf -y install nginx; systemctl enable --now nginx4# firewall-cmd --permanent --add-service=http; firewall-cmd --reload# ss -tlnp | grep :805LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1204,fd=6))1A static addresssaved in the profile2A permanent namehostnamectl3A local namechecked the way programs do4A serviceinstalled and started5Open and listeningfirewall and ss agreeLab: give a server a fixed address and name, publish a web service, and prove it is reachable.You need a VM; a second VM lets you test from outside. Press the down arrow for the steps.

A static IPv4 and IPv6 address, a hostname, a hosts entry, a web service through the firewall, a static route and a port forward.

Lab · Going deeper

A fixed address, a name and an open port

Use a RHEL-family VM (AlmaLinux or Rocky are free). Keep its console open: a wrong address can cut your SSH session. Replace the example addresses with ones on your own network.

Steps

  1. Read the current state. ip -br addr; ip route; nmcli dev; nmcli con show; cat /etc/resolv.conf. Note the connection name, address, gateway and DNS.
  2. Make it static. nmcli con mod "<name>" ipv4.method manual ipv4.addresses <addr>/24 ipv4.gateway <gw> ipv4.dns <dns>; nmcli con up "<name>". Reconnect if you were moved.
  3. Add IPv6. nmcli con mod "<name>" ipv6.method manual ipv6.addresses 2001:db8:10::21/64; nmcli con up; ip -6 addr.
  4. Name it. hostnamectl set-hostname web01.lab; add a line for another machine to /etc/hosts; compare getent hosts with dig for the same name.
  5. Publish a service. Install and enable nginx. curl -I http://localhost works; from the second VM, curl -I http://<addr> should time out or be refused.
  6. Open the firewall. firewall-cmd --permanent --add-service=http; firewall-cmd --reload; --list-all. The outside curl now works.
  7. Bind it locally. Change nginx to listen 127.0.0.1:80, reload, and watch ss -tlnp and the outside curl. Change it back.
  8. Route and forward. Add a static route with nmcli con mod "<name>" +ipv4.routes "10.20.0.0/16 <gw>"; ip route get 10.20.4.1. Add a forward port 8080 to 80 with firewall-cmd and test it from outside.
  9. Prove it. Reboot. The address, IPv6, hostname, route, firewall rules and nginx must all come back.
Check: After a reboot, ip -br addr shows both addresses, hostnamectl shows web01.lab, and curl -I http://<addr> and http://<addr>:8080 from another machine both return HTTP/1.1 200 OK.

Lesson complete

You finished “Networking”

Nicely done.

Open in a new tab