The problem
Start with facts: who is affected, since when, and what changed
Step one of the troubleshooting method: collect the facts. Everyone, since nine ten, works on 4G, and the server rebooted overnight.
It is ten past nine on a Monday, and the whole office says the company site will not load. Someone tries it on a phone, using mobile data, and it loads at once. And one more fact, the most useful of all: last night the office server was patched and rebooted. Before touching anything, we write these facts down.
Halving the path
One clue removes half the path: the fault is inside the office
The phone proves the internet and the site work, and every user fails, so look at what all office traffic shares.
The phone reached the same site over a different network. So the site is up, its address is right, and the internet in between works. That leaves the office half: the laptop, the switch, the router and firewall, and any service every machine relies on. And since every user fails, it is something they all share.
Testing each layer
Test one layer at a time; the first failure is where to look
Link, address, gateway and internet all pass; the name lookup is the first test to fail.
At one laptop we climb the layers. The link is up. The address came from DHCP and looks right. The gateway answers in 1 millisecond, and an address on the internet answers in 14. Layers one to three work all the way out. Then the name lookup times out, and the web test fails because it cannot turn the name into an address. Two failures, one cause.
The cause
Only name lookups fail: the office resolver stopped answering
The office DNS resolver 192.168.1.2 is down; a public resolver answers at once. The 02:14 reboot left the DNS service off.
Asking the office resolver gets no answer at all. Asking a public resolver directly gets the right address in 9 milliseconds, so the name is fine. The problem is the resolver the laptops were told to use. On the server, the DNS service is inactive, and it is not enabled at boot. So when the server rebooted at 02:14, DNS did not come back. Laptops kept cached answers for a while, which is why the complaints only started in the morning.
The fix
Fix the cause, then remove the single point of failure
Start and enable the service, then add a second resolver and hand both out in DHCP; laptops pick it up within 4 hours.
Before: every laptop uses one resolver, and it is down. After: we start the DNS service and enable it at boot, so names work again at once. Then we add a second resolver on another machine and give out both in DHCP. A laptop renews its lease halfway through, so with an 8-hour lease, every laptop has both resolvers within 4 hours, or straight away after a renew.
The trade-off
Every fix gives something up; choose, and write down why
A public resolver cannot answer internal names; a router-only resolver is one more single box. A second internal resolver costs one more service to run.
A public resolver as the backup is free and always up, but it knows nothing about internal names, and every lookup leaves the office. A second internal resolver answers everything and survives a reboot of the first, at the cost of one more thing to patch. A resolver only on the router is again a single box. We choose the second internal resolver, on a different machine, and write down why.
The method
The method is a ladder: ask what changed, then climb one layer at a time
Ask what changed, test link, switch, address, route, names and port in turn, then fix, verify and document. Each step links to its lesson.
Here is the whole method as one ladder. Ask what changed. Then the link and cable, the switch and VLAN, the address and gateway, the route out, names and DHCP, and the port and TLS. Then fix it, check it, and write it down. Each step opens the lesson that teaches it.
A real case
At Meta in 2021, the same symptom hid a much deeper fault
At Meta, a backbone outage made the DNS servers withdraw their BGP routes. DNS failing is a symptom; keep climbing to the cause.
On the fourth of October 2021, a command run during routine maintenance took down Meta’s backbone, the network between its data centres. Meta’s DNS servers are built to withdraw their routes when they cannot reach the data centres. They did, and the rest of the internet could no longer find facebook dot com. The same loss of DNS broke the tools engineers use to investigate, so people had to go to the data centres in person. The symptom was the one we saw in our office: names that would not resolve. The cause was much deeper. DNS failing is a symptom; keep climbing until you reach the cause.
Lesson complete
You finished “The site will not load”
Nicely done.