Method

Fix faults with a method, not a hunch

1Identify the problem2Establish a theoryof probable cause3Test the theory4Establish a planof action5Implement or escalate6Verify, and preventa repeat7Document findings andoutcometheory wrong? form a new oneStep 1 means: gather information, question users, find symptoms, ask what changed, copy the problem, take one problem at a time.Fix faults with a method, not a hunch: gather facts, test one theory at a time, then verify and write it down.Most outages follow a change, so “what changed?” is the most useful first question. Testing a theory should be cheap andreversible. Before you implement, know the rollback and the people affected; escalate when the fix is outside your authority.Documentation turns today’s fault into tomorrow’s two-minute fix.

The troubleshooting method: identify, theorise, test, plan, implement or escalate, verify and prevent, document.

Layers

Work through the layers; a ping splits the problem in half

SymptomCheck with7 Applicationname does not resolve, app errornslookup, dig, curl4 Transportport blocked or closednc -vz, Test-NetConnection, ACLs3 Networkwrong address, mask, gateway or routeping, traceroute, show ip route2 Data linkwrong VLAN, STP block, MAC tableshow vlan, show mac address-table1 Physicalno link, bad cable, errors, PoElink light, show interfaces, testerbottom upWork through the layers; a ping from the middle splits the problem in half.Bottom up starts at the cable and suits new installs; top down starts at the application and suits one broken app.Divide and conquer starts at layer 3: if the ping works, look up the stack; if not, look down. Change one thing at a time,and compare with something that works, such as the next desk, the same app from another site, or yesterday’s baseline.

Troubleshoot bottom up, top down or divide and conquer: a ping at layer 3 tells you whether to look up or down.

Ping and traceroute

Traceroute maps the path by making each router answer in turn

YouTTL 110.1.20.11 mstime exceededTTL 2203.0.113.19 mstime exceededTTL 3198.51.100.914 mstime exceededTTL 4192.0.2.33* * *no answerTTL 5203.0.113.8038 msecho reply$ traceroute 203.0.113.80 1  10.1.20.1  1.2 ms  1.4 ms  1.1 ms 2  203.0.113.1  9.2 ms  9.4 ms  9.1 ms 3  198.51.100.9  14.2 ms  14.4 ms  14.1 ms 4  *  *  * 5  203.0.113.80  38.2 ms  38.4 ms  38.1 msTraceroute sends packets with TTL 1, 2, 3…; each router that drops one reports back, mapping the path hop by hop.Every router lowers the TTL by one and, at zero, returns ICMP time exceeded. Times grow with distance. A hop showing * * * butlater hops answering is only a router that will not reply; stars to the end mark where packets die. Windows uses tracert.

Ping proves reachability; traceroute raises the TTL by one each time, so each router answers with time exceeded.

Ping and traceroute · Going deeper

Reading ping results

What ping saysMeansLook atReply from 10.9.0.80 … TTL=62reachable, 2 routers away (64 − 62)nothing at layer 3Request timed outno answer: down, filtered or no route backfirewalls, the return routeDestination host unreachableno path, or no ARP replygateway, mask, ARPTTL expired in transitthe packet went round in a looprouting tablesCisco: !!!!! ..... UUUUUreply · timeout · unreachableC:\> ipconfig /all       $ ip addr; ip route       C:\> arp -aC:\> ping 10.1.20.1     (gateway)     ping 10.9.0.53 (DNS)     ping www.example.com (name)Ping in widening circles: yourself, your gateway, a remote address, then a name.Each step that works clears a layer: 127.0.0.1 proves the stack, the gateway proves the LAN, a remote IP proves routing, and aname proves DNS. Many hosts block ICMP, so no reply is not proof of a fault; test the real port too.

Request timed out, destination host unreachable and TTL expired in transit each point somewhere different.

Interface counters

Interface counters tell cabling faults from duplex faults

GigabitEthernet1/0/5 is up, line protocol is up  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX     9,210,442 packets input, 7,402,118,330 bytes     4,812 input errors, 4,812 CRC, 0 frame, 0 overrun     0 runts, 0 giants, 0 throttles     0 collisions, 0 late collision, 0 deferred     12 output drops (total output drops)Status, then causeup / upworkingdown / downcable, far end, speedup / downlayer 2 protocol faultadministratively downshutdown in the configerr-disabledport security, BPDU guardCRC rate: 4,812 ÷ 9,210,442 = 0.052%no collisions, so not duplex: cable or EMIInterface counters name the layer 1 and 2 fault: CRC errors point to cabling, late collisions to a duplex mismatch.CRC errors with no collisions mean frames arrive damaged: a bad cable, a bad port or interference. Late collisions and runtsmean one side runs half duplex. Giants suggest an MTU mismatch; output drops mean congestion. Clear the counters, wait, andlook again: counters that do not grow are history.

CRC errors (0.052% here) mean damaged frames, so check the cable; late collisions mean a duplex mismatch.

Packet capture

A capture shows what really crossed the wire

Case 1: a firewall drops the SYN silently10.000  10.1.20.15.51234 > 10.9.0.80.443: Flags [S], seq 3201554810, win 6424011.000  10.1.20.15.51234 > 10.9.0.80.443: Flags [S], seq 3201554810, win 6424013.000  10.1.20.15.51234 > 10.9.0.80.443: Flags [S], seq 3201554810, win 6424017.000  10.1.20.15.51234 > 10.9.0.80.443: Flags [S], seq 3201554810, win 64240SYN resent after 1 s, 2 s, 4 s: the retry timer doubles. Nothing comes back, so something in the path drops it.Case 2: the server answers, but nothing listens on the port10.000  10.1.20.15.51240 > 10.9.0.80.8443: Flags [S], seq 118022341110.002  10.9.0.80.8443 > 10.1.20.15.51240: Flags [R.], ack 1180223412A capture shows what really crossed the wire: repeated SYNs mean a silent drop, a reset means a closed port.tcpdump and Wireshark decode every header. Capture where you can see both directions: on the host, with a switch SPAN(port mirror) or with a network tap. Filter tightly (host, port), and remember captures can hold passwords and personal data.

In a packet capture, repeated SYNs with no answer mean a silent drop; an immediate RST means nothing listens on the port.

Logs and monitoring

Logs and monitoring tell you what happened before you arrived

0Emergency1Alert2Critical3Error4Warning5Notice6Informational7Debuggingtrap warning: sends 0–4*Nov 15 09:14:07.311: %LINEPROTO-5-UPDOWN:  Line protocol on Interface Gi1/0/5,  changed state to down*Nov 15 09:14:09.002: %OSPF-5-ADJCHG: Process 1,  Nbr 2.2.2.2 on Gi0/0/0 from FULL to DOWNfacility - severity - mnemonic: messageSyslogevents as they happen, UDP 514SNMPcounters polled; traps on changeNetFlow, IPFIXwho talked to whom, how muchBaselinenormal levels, so spikes stand outSyslog severities run from 0, Emergency, to 7, Debugging; a logging level sends that level and everything more severe.So logging trap 4 sends 0 to 4. Messages carry a timestamp, which is why NTP matters, and a facility-severity-mnemonic code.Central logging, SNMP polling and flow records together show what happened, how busy things are and who is talking.

Syslog severity runs from 0 (Emergency) to 7 (Debugging); logging trap 4 sends 0–4. SNMP, NetFlow and a baseline complete the picture.

Symptoms

Each symptom points to a short list of causes

SymptomLikely causePing by IP works, by name failsDNS server or recordAddress 169.254.x.xDHCP unreachable or scope exhausted“Duplicate address” warningIP conflict: static inside the DHCP rangeSame subnet works, others failwrong gateway or maskOne VLAN’s users all failtrunk missing the VLAN, SVI downSome sites work, one does notACL or firewall rule, asymmetric routeSmall pings pass, big transfers hangMTU black hole, often a tunnelTTL expired, repeating hopsrouting loopEverything slow, links lit solidbroadcast storm or STP loopNew phones dark on one switchPoE budget exceededEach symptom points to a short list of causes; learn the pairs and the first test for each.Many faults are configuration: a wrong VLAN, a missing route, an ACL line in the wrong place. Performance faults show up aslatency, jitter, packet loss or bottlenecks; compare against the baseline before blaming the network.

Know the pairs: name fails but IP works means DNS; 169.254 means DHCP; repeating hops mean a loop; small pings pass but big transfers hang means MTU.

Symptoms · Going deeper

Hardware and software tools

ToolUseCable tester, certifierwiring faults, length, category performanceToner and probefind which cable is which in a bundleTDR, OTDRdistance to a break in copper or fibreLight meter, visual fault locatorfibre power in dBm; a red laser shows breaksWi-Fi and spectrum analyserssignal, channels and non-Wi-Fi interferenceTap, SPAN port, protocol analysercapture traffic without changing itnmap, netstat, ssopen ports, listening services, connectionsCDP, LLDPwhich device and port is on the far endHardware tools find physical faults that no command can see; software tools show what the network is doing.

Cable testers, toner probes, OTDR, light meters, Wi-Fi analysers, taps and SPAN ports, nmap and LLDP.

Lesson complete

You finished “Troubleshooting”

Nicely done.

Open in a new tab