System Monitoring Fundamentals

Overview

• What you’ll learn: The fundamentals of system monitoring on Linux, including CPU, memory, disk, and network metrics, and how to use essential monitoring utilities such as top, vmstat, iostat, and sar.
• Prerequisites: Linux Fundamentals (Module 1), basic command-line proficiency
• Estimated reading time: 18 minutes

Introduction

System monitoring is the practice of continuously observing the health and performance of a computing system. In production Linux environments, monitoring is not optional — it is the foundation on which incident response, capacity planning, and performance tuning are built. Without monitoring, administrators are essentially flying blind, reacting to outages instead of preventing them.

Effective monitoring begins with understanding what to measure. The four golden signals of system health are CPU utilization, memory consumption, disk I/O throughput, and network bandwidth. Each of these metrics tells a story about how the system is performing and where bottlenecks may be forming.

In this lesson, we will explore the theory behind system monitoring, learn to read key metrics from the Linux kernel’s virtual filesystems, and practice using the most important command-line monitoring tools available on Ubuntu Server. By the end, you will be able to diagnose common performance issues and establish a baseline for normal system behavior.

Why Monitoring Matters

Monitoring serves several critical purposes in a production environment:

• Early warning: Detecting anomalies before they escalate into outages.
• Capacity planning: Understanding resource trends so you can scale infrastructure proactively.
• Root cause analysis: Providing historical data that helps pinpoint the cause of incidents.
• SLA compliance: Proving that systems meet agreed-upon uptime and performance targets.

The cost of downtime in enterprise environments can range from thousands to millions of dollars per hour. Investing in monitoring infrastructure pays for itself many times over by reducing mean time to detection (MTTD) and mean time to recovery (MTTR).

The Four Golden Metrics

CPU Utilization

CPU utilization measures the percentage of time the processor spends executing tasks rather than sitting idle. On Linux, CPU time is divided into several categories: user time (running application code), system time (running kernel code), I/O wait (blocked waiting for disk), and idle time.

# View real-time CPU usage with top
$ top -bn1 | head -5
top - 14:23:01 up 45 days,  3:12,  2 users,  load average: 0.42, 0.38, 0.35
Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.2 us,  2.1 sy,  0.0 ni, 91.8 id,  0.7 wa,  0.0 hi,  0.2 si,  0.0 st

# View per-CPU breakdown
$ mpstat -P ALL 1 1
Linux 5.15.0-91-generic (ubuntu-server)   03/02/2026  _x86_64_

02:23:01 PM  CPU  %usr  %nice  %sys  %iowait  %irq  %soft  %steal  %idle
02:23:02 PM  all  5.20   0.00  2.10     0.70  0.00   0.20    0.00  91.80
02:23:02 PM    0  6.00   0.00  3.00     1.00  0.00   0.00    0.00  90.00
02:23:02 PM    1  4.00   0.00  1.00     0.00  0.00   1.00    0.00  94.00

Memory Consumption

Linux memory management is nuanced. The kernel aggressively caches disk data in RAM, so “used” memory often includes cache that can be reclaimed. The key metric is available memory — the amount that can be allocated to new processes without swapping.

# View memory summary
$ free -h
              total   used   free   shared  buff/cache  available
Mem:           15Gi   4.2Gi  1.8Gi   256Mi      9.2Gi     10.5Gi
Swap:         2.0Gi     0B   2.0Gi

# Detailed memory info from /proc
$ cat /proc/meminfo | head -10
MemTotal:       16384000 kB
MemFree:         1884160 kB
MemAvailable:   11059200 kB
Buffers:          524288 kB
Cached:          8921088 kB
SwapTotal:       2097152 kB
SwapFree:        2097152 kB

Disk I/O

Disk I/O performance is measured in terms of throughput (MB/s), IOPS (I/O operations per second), and latency (time per operation). High I/O wait in the CPU metrics often indicates a disk bottleneck.

# Monitor disk I/O with iostat
$ iostat -xz 1 1
Device  r/s    w/s    rkB/s   wkB/s  rrqm/s  wrqm/s  %rrqm  %wrqm  r_await  w_await  aqu-sz  rareq-sz  wareq-sz  svctm  %util
sda     12.50  35.20  200.00  560.00  0.50    8.30    3.85   19.09  1.20     2.50     0.08   16.00     15.91     0.45   2.14

# Check disk space usage
$ df -h
Filesystem      Size  Used  Avail  Use%  Mounted on
/dev/sda1        50G   22G    26G   46%  /
/dev/sda2       200G   85G   105G   45%  /data

Network Bandwidth

Network monitoring tracks the volume and rate of data transmitted and received across network interfaces. Saturated network links cause latency spikes and packet loss.

# View network interface statistics
$ ip -s link show eth0
2: eth0:  mtu 1500 qdisc fq_codel state UP
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped
    8543210    65432    0       0
    TX: bytes  packets  errors  dropped
    3210987    43210    0       0

# Real-time bandwidth monitoring with nload
$ sudo apt install nload
$ nload eth0

Essential Monitoring Tools

top and htop

The top utility provides a real-time, dynamic view of running processes sorted by resource consumption. Its enhanced counterpart htop offers a more user-friendly interface with color coding, mouse support, and easier process management.

# Install htop
$ sudo apt install htop

# Run htop with tree view
$ htop --tree

# Key shortcuts in htop:
# F5 — tree view
# F6 — sort by column
# F9 — send signal to process
# F10 — quit

vmstat

The vmstat (virtual memory statistics) command reports information about processes, memory, paging, block I/O, traps, and CPU activity. It is particularly useful for spotting memory pressure and swapping.

# Report every 2 seconds, 5 iterations
$ vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1884160 524288 8396800  0    0    12    35  250  480  5  2 92  1  0
 0  0      0 1883904 524288 8396800  0    0     0    28  230  450  3  1 95  1  0

sar (System Activity Reporter)

The sar command, part of the sysstat package, collects and reports system activity data. Unlike top and vmstat, sar stores historical data, making it invaluable for trend analysis and post-incident investigation.

# Install sysstat and enable data collection
$ sudo apt install sysstat
$ sudo systemctl enable --now sysstat

# View CPU usage for today
$ sar -u
Linux 5.15.0-91-generic (ubuntu-server)   03/02/2026

12:00:01 AM  CPU  %user  %nice  %system  %iowait  %steal  %idle
12:10:01 AM  all   4.52   0.00    1.87     0.63    0.00   92.98
12:20:01 AM  all   3.98   0.00    1.56     0.45    0.00   94.01

# View memory usage history
$ sar -r

# View disk I/O history
$ sar -d

# View network throughput history
$ sar -n DEV

dstat

The dstat command combines the functionality of vmstat, iostat, and netstat into a single, color-coded display that updates in real time.

# Install dstat
$ sudo apt install dstat

# Show CPU, disk, net, and memory in one view
$ dstat -cdnm 5
----total-cpu-usage---- -dsk/total- -net/total- ------memory-usage-----
usr sys idl wai stl| read  writ| recv  send| used  free  buff  cach
  5   2  92   1   0| 200k  560k|  45k  32k| 4.2G 1.8G  512M 8.2G

Reading /proc and /sys

The Linux kernel exposes real-time system information through two virtual filesystems. The /proc filesystem provides process and kernel information, while /sys exposes device and driver data. Monitoring tools read from these filesystems internally, but understanding them directly gives you deeper diagnostic capabilities.

# Load averages (1, 5, 15 minute)
$ cat /proc/loadavg
0.42 0.38 0.35 1/187 4523

# CPU information
$ cat /proc/cpuinfo | grep "model name" | head -1
model name	: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

# Current disk I/O stats
$ cat /proc/diskstats | grep sda

# Network interface counters
$ cat /proc/net/dev

Building a Monitoring Baseline

A baseline is a record of normal system behavior collected over a representative period — typically one to four weeks. Without a baseline, it is impossible to distinguish abnormal behavior from normal fluctuations.

Steps to establish a baseline:

1. Enable sysstat data collection (already done above).
2. Let the system run through typical workload cycles (daily, weekly).
3. Use sar to extract averages and peaks for CPU, memory, disk, and network.
4. Document the results and set alert thresholds slightly above peak values.

# Generate a summary report for a specific day
$ sar -u -f /var/log/sysstat/sa01
$ sar -r -f /var/log/sysstat/sa01
$ sar -d -f /var/log/sysstat/sa01

# Export data to CSV for analysis
$ sadf -d /var/log/sysstat/sa01 -- -u > cpu_report.csv

Key Takeaways

• The four golden metrics — CPU, memory, disk I/O, and network — form the foundation of system monitoring.
• top/htop provide real-time process-level views; vmstat and iostat give system-level summaries.
• sar from the sysstat package records historical data essential for trend analysis and post-incident investigation.
• Linux exposes all metrics through /proc and /sys virtual filesystems; monitoring tools are wrappers around these.
• Always establish a performance baseline before setting alert thresholds.
• The “available” field in free -h is the correct metric for usable memory, not “free”.

What’s Next

In the next lesson, we will explore Nagios Core Monitoring — a comprehensive open-source monitoring framework that builds on these fundamentals to provide centralized, automated monitoring with alerting and visualization for entire infrastructure environments.

# 使用 top 查看即時 CPU 使用率
$ top -bn1 | head -5

# 查看每個 CPU 的分解
$ mpstat -P ALL 1 1

# 查看記憶體摘要
$ free -h

# 從 /proc 查看詳細記憶體資訊
$ cat /proc/meminfo | head -10

# 使用 iostat 監控磁碟 I/O
$ iostat -xz 1 1

# 檢查磁碟空間使用率
$ df -h

# 查看網路介面統計
$ ip -s link show eth0

$ sudo apt install htop
$ htop --tree

$ vmstat 2 5

$ sudo apt install sysstat
$ sudo systemctl enable --now sysstat
$ sar -u
$ sar -r
$ sar -n DEV

$ cat /proc/loadavg
$ cat /proc/cpuinfo | grep "model name" | head -1
$ cat /proc/net/dev

$ top -bn1 | head -5
$ mpstat -P ALL 1 1

$ free -h
$ cat /proc/meminfo | head -10

$ iostat -xz 1 1
$ df -h

$ ip -s link show eth0

$ sudo apt install htop
$ htop --tree

$ vmstat 2 5

$ sudo apt install sysstat
$ sudo systemctl enable --now sysstat
$ sar -u
$ sar -r
$ sar -n DEV

$ cat /proc/loadavg
$ cat /proc/cpuinfo | grep "model name" | head -1
$ cat /proc/net/dev