The problem

Start with facts: who is affected, since when, and what changed

Office192.168.1.0/2440 staffSwitchRouter, firewall192.168.1.1Office serverfiles, internal DNSInternetwww.example.com203.0.113.80✗ will not loadPhone on 4G: loadsFactsSince 09:10, all 40 staffLast night: server patchedand rebooted at 02:14Start with facts, not a guess: who is affected, since when, and what changed.At 09:10 on a Monday, all 40 people in the office report that the company site will not load. The same page opens on a phoneusing mobile data. One more fact matters most: the office server was patched and rebooted overnight. Step 1 of the methodis to collect exactly these facts before touching anything.

Step one of the troubleshooting method: collect the facts. Everyone, since nine ten, works on 4G, and the server rebooted overnight.

Halving the path

One clue removes half the path: the fault is inside the office

LaptopSwitchRouterInternetSitePhone on 4Gproved working by the phonethe fault is somewhere in the officeOne clue removes half the path: the site and the internet work, so the fault is inside the office.The phone reaches the same site over a different network, so the web server, its address and the internet are fine.Every user fails, so it is not one laptop or one cable. What is left is something all office traffic shares: the switch,the router and firewall, or a service every machine depends on.

The phone proves the internet and the site work, and every user fails, so look at what all office traffic shares.

Testing each layer

Test one layer at a time; the first failure is where to look

LayerTestResult1 Linklink light, ip linkup, 1000 Mb/s, full duplex2 Addressipconfig /all192.168.1.57/24, gateway 192.168.1.1, DNS 192.168.1.23 Gatewayping 192.168.1.1reply in 1 ms3 Internetping 8.8.8.8reply in 14 ms7 Namesnslookup www.example.comtimed out: no answer from 192.168.1.24–7 Webcurl -I https://www.example.comcould not resolve host✓✓✓✓✗✗first failureTest one layer at a time from the cable up; the first test that fails is where to look.The link is up, the laptop has a correct address from DHCP, the gateway answers and so does an address on the internet.So layers 1 to 3 work all the way out. The first failure is the name lookup, and the web test fails only because it needsa name first. Two failures, one cause.

Link, address, gateway and internet all pass; the name lookup is the first test to fail.

The cause

Only name lookups fail: the office resolver stopped answering

On a laptop$ nslookup www.example.com;; connection timed out; no servers;; could be reached (192.168.1.2)$ nslookup www.example.com 1.1.1.1Name:    www.example.comAddress: 203.0.113.80     (9 ms)On the office server, 192.168.1.2$ systemctl status named   Active: inactive (dead)$ systemctl is-enabled named   disabled$ last reboot | head -1   reboot  ...  Mon 02:14Only name lookups fail: the office DNS server stopped answering when the server rebooted.Asked directly, a public resolver answers in 9 ms with the right address, so the name and the site are fine. The resolver everylaptop was given, 192.168.1.2, does not answer at all. On that server the DNS service is not running and is not enabled at boot, so the02:14 reboot left it off. Laptops kept cached answers for a while, which is why complaints started hours later.

The office DNS resolver 192.168.1.2 is down; a public resolver answers at once. The 02:14 reboot left the DNS service off.

The fix

Fix the cause, then remove the single point of failure

BeforeLaptops192.168.1.2not runningDHCP option 6 (DNS): 192.168.1.2AfterLaptops192.168.1.2192.168.1.3second resolverDHCP option 6 (DNS): 192.168.1.2, 192.168.1.3Fix the cause first, then remove the single point of failure that made one service stop the whole office.Start the DNS service and enable it at boot: names work again at once. Then add a second resolver on another machine andhand both out by DHCP. Laptops pick up the new list when they renew their lease, at half its length: within 4 hours for an8-hour lease, or at once with ipconfig /renew. Then document the cause and the fix.

Start and enable the service, then add a second resolver and hand both out in DHCP; laptops pick it up within 4 hours.

The trade-off

Every fix gives something up; choose, and write down why

Internal namesSurvives the serverQueries stay insideNothing new to runPublic resolver as the second✗✓✗✓Second internal resolver✓✓✓✗Resolver on the router only✓✗✓✓Chosen: a second internal resolver on a different machine, so one reboot cannot take out both.Every fix gives something up; choose on what the office needs, and write the reason down.A public resolver is free and always up, but it cannot answer for internal names such as the file server, and every lookupleaves the building. A resolver only on the router is one more single box. A second internal resolver answers everythingand survives a reboot, at the cost of one more service to patch and watch.

A public resolver cannot answer internal names; a router-only resolver is one more single box. A second internal resolver costs one more service to run.

The method

The method is a ladder: ask what changed, then climb one layer at a time

1 Ask what changedLesson 162 Link and cableLesson 23 Switch and VLANLesson 114 Address, mask, gatewayLesson 55 Route outLesson 66 Names and DHCPLesson 107 Port and TLSLesson 98 Fix, verify, documentLesson 16Each step opens the lesson that teaches it.The method is a ladder: ask what changed, then climb one layer at a time, and finish by writing it down.This case used every rung. Most office faults stop at one of them: a cable, a VLAN, an address, a route, a name or a port.The order matters less than the habit: test one thing, read the result, and let the result choose the next test.

Ask what changed, test link, switch, address, route, names and port in turn, then fix, verify and document. Each step links to its lesson.

A real case

At Meta in 2021, the same symptom hid a much deeper fault

A commandmeant to measurebackbone capacityBackbone downdata centrescut offDNS withdrawsBGP routes pulledby a health checkNames failfacebook.comunreachableTools fail tooengineers goon siteOur officeone service did not start → names fail → the site "is down"Meta, 2021the backbone went down → names fail → every app "is down"At Meta in October 2021 the same symptom, names that would not resolve, came from a much deeper fault.A command during routine maintenance took down the backbone; an audit tool meant to stop it had a bug. The DNS servers, built towithdraw their BGP routes when they cannot reach the data centres, did so, and the world could no longer find Meta. Loss ofDNS also broke the tools engineers use to fix outages. Lesson: DNS failing is a symptom; keep climbing until you reach the cause.

At Meta, a backbone outage made the DNS servers withdraw their BGP routes. DNS failing is a symptom; keep climbing to the cause.

Lesson complete

You finished “The site will not load”

Nicely done.

Open in a new tab