Method
Fix faults with a method, not a hunch
The troubleshooting method: identify, theorise, test, plan, implement or escalate, verify and prevent, document.
First, find out exactly what is wrong: who is affected, since when, and above all what changed. Then form a theory of the likely cause and test it. If the test proves it wrong, form another. Once a theory holds, plan the fix, including how to undo it. Implement it, or escalate if it is not yours to change. Check that everything works, prevent a repeat, and write down what you found.
Layers
Work through the layers; a ping splits the problem in half
Troubleshoot bottom up, top down or divide and conquer: a ping at layer 3 tells you whether to look up or down.
Start in the middle. Ping the far end. If it answers, layers one to three are fine, so the fault is higher up. Is the port open? Does the name resolve? If the ping fails, look down: the address, mask and gateway; the VLAN and spanning tree; and finally the cable and the interface counters.
Ping and traceroute
Traceroute maps the path by making each router answer in turn
Ping proves reachability; traceroute raises the TTL by one each time, so each router answers with time exceeded.
Traceroute sends a packet that may cross only one router. That router drops it and says so, revealing itself. Then two routers, then three. One hop shows stars: it simply does not reply, and the hops after it do, so nothing is wrong there. Finally the destination answers. If the stars continued to the end, that is where packets die.
Ping and traceroute · Going deeper
Reading ping results
Request timed out, destination host unreachable and TTL expired in transit each point somewhere different.
Interface counters
Interface counters tell cabling faults from duplex faults
CRC errors (0.052% here) mean damaged frames, so check the cable; late collisions mean a duplex mismatch.
show interfaces starts with two words: is the port up, and is the line protocol up. Down and down is usually the cable or the far end. Administratively down means someone typed shutdown. Then the counters. This port has 4,812 CRC errors out of 9,210,442 packets: frames arriving damaged. There are no collisions, so it is not duplex. Replace the cable, then clear the counters and watch.
Packet capture
A capture shows what really crossed the wire
In a packet capture, repeated SYNs with no answer mean a silent drop; an immediate RST means nothing listens on the port.
The client sends a SYN to port four four three. Nothing comes back. It tries again after one second, two, four. Something is silently dropping it: a firewall or a missing return route. In the second capture the server answers at once, with a reset. The path is fine; nothing is listening on port eighty-four forty-three.
Logs and monitoring
Logs and monitoring tell you what happened before you arrived
Syslog severity runs from 0 (Emergency) to 7 (Debugging); logging trap 4 sends 0–4. SNMP, NetFlow and a baseline complete the picture.
Every log message has a severity from zero, emergency, to seven, debugging. Set a level, and the device sends that level and everything more severe: level 4, warnings, sends zero to 4. Here, an interface went down, and two seconds later the OSPF neighbour behind it dropped. Logs say what happened; SNMP says how busy things are; flow records say who is talking to whom; a baseline says what normal looks like.
Symptoms
Each symptom points to a short list of causes
Know the pairs: name fails but IP works means DNS; 169.254 means DHCP; repeating hops mean a loop; small pings pass but big transfers hang means MTU.
With experience, symptoms point straight at causes. A name that will not resolve while the address works is DNS. An address starting 169.254 means DHCP never answered. Users in one VLAN all failing points at the trunk. Small pings passing while large transfers hang is an MTU problem, often a tunnel. And everything slow with every link light solid is a loop.
Symptoms · Going deeper
Hardware and software tools
Cable testers, toner probes, OTDR, light meters, Wi-Fi analysers, taps and SPAN ports, nmap and LLDP.
Lesson complete
You finished “Troubleshooting”
Nicely done.