Network Troubleshooting Tools

Overview

• What you’ll learn: A systematic approach to network troubleshooting, and how to use ping, traceroute, mtr, ss, netstat, curl, tcpdump, nmap, and tshark/Wireshark to diagnose connectivity, performance, and security issues.
• Prerequisites: Lesson 19 – Firewall with ufw and iptables
• Estimated reading time: 25 minutes

Introduction

Network problems are among the most common and frustrating issues in system administration. A server that cannot reach the Internet, a service that times out, or a mysterious packet loss can bring entire systems to a halt. The key to effective troubleshooting is a systematic, layer-by-layer approach combined with the right tools.

Linux provides an exceptionally rich set of network diagnostic tools, from simple utilities like ping to powerful packet analyzers like tcpdump. Mastering these tools allows you to quickly identify whether a problem is at the physical layer, the network layer, the transport layer, or the application layer.

In this lesson, you will build a comprehensive troubleshooting methodology and learn to use each tool effectively with real-world examples.

Systematic Troubleshooting Approach

When diagnosing network issues, work from the bottom of the OSI model upward. This layer-by-layer approach ensures you do not waste time on application-layer debugging when the problem is actually a broken cable or a misconfigured route.

• Layer 1 (Physical): Is the cable connected? Is the interface up? Check ip link show.
• Layer 2 (Data Link): Can you see the MAC address of the gateway? Check ip neigh show.
• Layer 3 (Network): Can you ping the gateway? Can you ping external IPs? Check ping and ip route.
• Layer 4 (Transport): Is the port open? Is the service listening? Check ss and nc.
• Layer 7 (Application): Is the application responding correctly? Check curl and application logs.

# Quick connectivity check script
#!/bin/bash
echo "=== Interface Status ==="
ip link show ens33 | grep -E "state (UP|DOWN)"

echo "=== IP Address ==="
ip addr show ens33 | grep "inet "

echo "=== Default Gateway ==="
ip route show default

echo "=== Gateway Reachability ==="
ping -c 2 -W 2 $(ip route show default | awk '{print $3}') 2>&1 | tail -3

echo "=== DNS Resolution ==="
dig +short +time=2 www.example.com

echo "=== External Connectivity ==="
ping -c 2 -W 2 8.8.8.8 2>&1 | tail -3

Ping, Traceroute, and MTR

ping tests basic IP connectivity by sending ICMP echo requests. traceroute reveals the path packets take to a destination, showing each hop. mtr combines both into a continuous, real-time display that is ideal for identifying intermittent packet loss.

# Basic ping — test connectivity and measure latency
$ ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=4.23 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=4.15 ms
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 4.123/4.189/4.234/0.041 ms

# Ping with specific packet size (test MTU issues)
$ ping -c 3 -s 1472 -M do 8.8.8.8

# Traceroute — show the path to destination
$ traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.1.1     1.234 ms  1.123 ms  1.089 ms
 2  10.0.0.1        5.678 ms  5.432 ms  5.321 ms
 3  * * *           (no response — firewall blocking ICMP)
 4  8.8.8.8        15.678 ms 15.432 ms 15.321 ms

# MTR — continuous traceroute with statistics
$ mtr -n -c 20 8.8.8.8
                         Packets               Pings
 Host                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1         0.0%    20    1.2   1.3   1.0   2.1   0.3
 2. 10.0.0.1            0.0%    20    5.4   5.6   5.1   7.2   0.5
 3. 172.16.0.1          5.0%    20   12.3  12.8  11.9  15.4   0.9
 4. 8.8.8.8             0.0%    20   15.2  15.5  14.8  17.1   0.6

# Install mtr if not available
$ sudo apt install -y mtr-tiny

Socket and Connection Analysis

The ss command (socket statistics) is the modern replacement for netstat. It displays detailed information about TCP connections, listening ports, UDP sockets, and Unix domain sockets.

# Show all listening TCP ports
$ ss -tlnp
State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
LISTEN  0       128     0.0.0.0:22           0.0.0.0:*          users:(("sshd",pid=1234,fd=3))
LISTEN  0       511     0.0.0.0:80           0.0.0.0:*          users:(("nginx",pid=5678,fd=6))
LISTEN  0       128     0.0.0.0:443          0.0.0.0:*          users:(("nginx",pid=5678,fd=7))

# Show all established connections
$ ss -tnp
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port   Process
ESTAB   0       0       192.168.1.10:22      192.168.1.5:54312   users:(("sshd",pid=9012,fd=4))

# Show socket summary statistics
$ ss -s
Total: 187
TCP:   12 (estab 3, closed 2, orphaned 0, timewait 1)

# Show connections to a specific port
$ ss -tnp dst :443

# Count connections per state
$ ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn
     45 ESTAB
     12 TIME-WAIT
      6 LISTEN
      3 CLOSE-WAIT

# Check if a specific port is reachable (with nc)
$ nc -zv 192.168.1.20 80
Connection to 192.168.1.20 80 port [tcp/http] succeeded!

# Test HTTP with curl
$ curl -I -s -o /dev/null -w "%{http_code} %{time_total}sn" https://example.com
200 0.234567s

Packet Capture with tcpdump

tcpdump is a powerful command-line packet analyzer. It captures packets on a network interface and displays them in real time or saves them to a file for later analysis.

# Capture all traffic on an interface
$ sudo tcpdump -i ens33 -c 10

# Capture traffic for a specific host
$ sudo tcpdump -i ens33 host 192.168.1.20

# Capture only TCP traffic on port 80
$ sudo tcpdump -i ens33 tcp port 80 -nn

# Capture DNS queries
$ sudo tcpdump -i ens33 udp port 53 -nn
10:15:01 IP 192.168.1.10.45678 > 8.8.8.8.53: 12345+ A? www.example.com.
10:15:01 IP 8.8.8.8.53 > 192.168.1.10.45678: 12345 1/0/0 A 93.184.216.34

# Save capture to a file (for Wireshark analysis)
$ sudo tcpdump -i ens33 -w /tmp/capture.pcap -c 1000

# Read a saved capture file
$ tcpdump -r /tmp/capture.pcap -nn | head -20

# Advanced filters — capture SYN packets only (new connections)
$ sudo tcpdump -i ens33 'tcp[tcpflags] & tcp-syn != 0' -nn

# Capture packets larger than 1000 bytes
$ sudo tcpdump -i ens33 greater 1000 -nn

Port Scanning with nmap

nmap (Network Mapper) is a versatile tool for network discovery and security auditing. It can scan for open ports, detect running services, and identify operating systems.

# Install nmap
$ sudo apt install -y nmap

# Basic TCP port scan
$ nmap 192.168.1.20
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

# Scan specific ports
$ nmap -p 22,80,443,3306 192.168.1.20

# Scan a range of ports
$ nmap -p 1-1024 192.168.1.20

# Service version detection
$ nmap -sV 192.168.1.20
PORT    STATE SERVICE VERSION
22/tcp  open  ssh     OpenSSH 8.9p1 Ubuntu 3ubuntu0.6
80/tcp  open  http    nginx 1.18.0
443/tcp open  ssl     nginx 1.18.0

# UDP scan (slower but important)
$ sudo nmap -sU -p 53,67,68,123 192.168.1.1

# Scan an entire subnet
$ nmap -sn 192.168.1.0/24
Nmap scan report for 192.168.1.1
Host is up (0.0012s latency).
Nmap scan report for 192.168.1.10
Host is up (0.00045s latency).

# OS detection (requires root)
$ sudo nmap -O 192.168.1.20

Key Takeaways

• Always troubleshoot systematically, working from the physical layer upward: interface status, IP configuration, gateway reachability, DNS resolution, then application connectivity.
• Use ping for basic reachability, traceroute for path analysis, and mtr for identifying intermittent packet loss at specific hops.
• The ss command is the modern tool for inspecting sockets and connections; combine it with nc and curl for port and service testing.
• tcpdump captures packets at the wire level — invaluable for diagnosing protocol-level issues; save captures to .pcap files for detailed Wireshark analysis.
• nmap discovers open ports, running services, and operating systems — essential for security auditing and verifying firewall configurations.

What’s Next

Congratulations on completing Module 2 — TCP/IP and Networking! You now have a solid foundation in network protocols, configuration, DNS, DHCP, firewalls, and troubleshooting. In Module 3, you will advance to Storage and File Systems, where you will learn about disk partitioning, LVM, file system types, and RAID configurations.