Overview

• What you’ll learn: How Linux logging works under systemd-journald and rsyslog, how to query and filter logs with journalctl, how to deploy Logwatch for automated log analysis, and best practices for log retention, rotation, and centralization.
• Prerequisites: System Monitoring Fundamentals (Lesson 61), systemd basics
• Estimated reading time: 18 minutes

Introduction

Logs are the narrative history of everything that happens on a Linux system. Every service startup, authentication attempt, error condition, and configuration change is recorded in log files. For system administrators, logs are the primary tool for troubleshooting, security auditing, and compliance reporting.

Modern Ubuntu Server systems use two parallel logging subsystems: systemd-journald, which captures structured log data from all systemd-managed services, and rsyslog, which provides traditional syslog-compatible logging to text files. Understanding both systems and how they interact is essential for effective log management.

In this lesson, we will master the journalctl command for querying the systemd journal, configure rsyslog for log forwarding, deploy Logwatch for automated daily log summaries, and implement log rotation policies to manage disk space.

The Linux Logging Architecture

On a systemd-based Ubuntu system, logging flows through several layers:

• systemd-journald: The primary log collector. It captures stdout/stderr from all systemd services, kernel messages, and syslog messages. The journal is stored in a binary format in /var/log/journal/ (persistent) or /run/log/journal/ (volatile).
• rsyslog: The traditional syslog daemon. It reads from the journal and writes categorized log files to /var/log/ (e.g., syslog, auth.log, kern.log).
• Application logs: Some applications write directly to their own log files (e.g., Apache logs in /var/log/apache2/, Nginx logs in /var/log/nginx/).

# Check if journald is running with persistent storage
$ systemctl status systemd-journald
$ ls -la /var/log/journal/

# Check if rsyslog is active
$ systemctl status rsyslog

# View the standard log files rsyslog creates
$ ls -la /var/log/syslog /var/log/auth.log /var/log/kern.log

Mastering journalctl

The journalctl command is the primary interface for querying the systemd journal. It supports powerful filtering by time, service unit, priority level, and many other fields.

# View all journal entries (newest last)
$ journalctl

# Follow new entries in real time (like tail -f)
$ journalctl -f

# Show entries from the current boot only
$ journalctl -b

# Show entries from the previous boot
$ journalctl -b -1

# Filter by systemd unit
$ journalctl -u nginx.service
$ journalctl -u ssh.service --since "1 hour ago"

# Filter by time range
$ journalctl --since "2026-03-01 00:00:00" --until "2026-03-01 23:59:59"
$ journalctl --since "yesterday" --until "today"

# Filter by priority (0=emerg through 7=debug)
$ journalctl -p err                  # Show only errors and above
$ journalctl -p warning..err         # Show warnings and errors

# Filter by process PID
$ journalctl _PID=1234

# Show kernel messages only
$ journalctl -k

# Output in JSON format for parsing
$ journalctl -u nginx.service -o json-pretty --no-pager | head -30

# Show disk usage of the journal
$ journalctl --disk-usage
Archived and active journals take up 256.0M in /var/log/journal.

Configuring journald

The journal is configured through /etc/systemd/journald.conf. Key settings control storage persistence, size limits, and retention policies.

# Edit journald configuration
$ sudo nano /etc/systemd/journald.conf

[Journal]
# Persistent storage (survives reboots)
Storage=persistent

# Maximum disk usage for journal files
SystemMaxUse=500M

# Maximum size of individual journal files
SystemMaxFileSize=50M

# How long to keep journal entries
MaxRetentionSec=30day

# Rate limiting (per-service)
RateLimitIntervalSec=30s
RateLimitBurst=10000

# Restart journald to apply changes
$ sudo systemctl restart systemd-journald

rsyslog Configuration

While journald captures everything, rsyslog provides text-based log files that are easier to process with traditional Unix tools and are required by many compliance frameworks.

# Main rsyslog configuration
$ cat /etc/rsyslog.conf

# Facility.Priority routing rules in /etc/rsyslog.d/50-default.conf
auth,authpriv.*          /var/log/auth.log
*.*;auth,authpriv.none   /var/log/syslog
kern.*                   /var/log/kern.log
mail.*                   /var/log/mail.log

# Forward logs to a remote syslog server
$ sudo tee /etc/rsyslog.d/60-remote.conf > /dev/null <<'EOF'
# Forward all logs to central syslog server via TCP
*.* @@logserver.example.com:514

# Forward only auth logs via UDP
auth,authpriv.* @logserver.example.com:514
EOF

$ sudo systemctl restart rsyslog

Log Rotation with logrotate

Without rotation, log files would grow indefinitely and eventually fill the disk. The logrotate utility automatically compresses, rotates, and removes old log files based on configurable policies.

# Global logrotate configuration
$ cat /etc/logrotate.conf

# Application-specific configuration
$ cat /etc/logrotate.d/rsyslog
/var/log/syslog
/var/log/mail.log
/var/log/kern.log
/var/log/auth.log
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

# Create a custom logrotate config for an application
$ sudo tee /etc/logrotate.d/myapp > /dev/null <<'EOF'
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data www-data
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}
EOF

# Test logrotate configuration (dry run)
$ sudo logrotate -d /etc/logrotate.d/myapp

# Force rotation for testing
$ sudo logrotate -f /etc/logrotate.d/myapp

Deploying Logwatch

Logwatch is an automated log analysis tool that parses system logs and generates human-readable summary reports. It is particularly useful for daily security reviews, as it highlights failed login attempts, service errors, disk usage warnings, and other notable events.

# Install Logwatch
$ sudo apt install -y logwatch

# Run Logwatch manually for today's logs
$ sudo logwatch --detail High --range today --output stdout

# Run for a specific date range
$ sudo logwatch --detail Med --range "between 2/28/2026 and 3/1/2026" --output stdout

# Generate an HTML report
$ sudo logwatch --detail High --range yesterday --output file 
    --filename /tmp/logwatch-report.html --format html

# Configure Logwatch defaults
$ sudo nano /etc/logwatch/conf/logwatch.conf
Output = mail
Format = html
MailTo = admin@example.com
MailFrom = logwatch@server.example.com
Detail = Med
Range = yesterday

# Logwatch runs automatically via daily cron
$ ls -la /etc/cron.daily/00logwatch

Practical Log Analysis Patterns

Combining journalctl with traditional text processing tools allows for powerful ad-hoc log analysis.

# Find failed SSH login attempts
$ journalctl -u ssh.service | grep "Failed password"

# Count failed logins by IP address
$ journalctl -u ssh.service --since "24 hours ago" --no-pager 
    | grep "Failed password" 
    | grep -oP 'from K[0-9.]+' 
    | sort | uniq -c | sort -rn | head -10

# Find out-of-memory (OOM) killer events
$ journalctl -k | grep -i "oom"

# Check for disk errors
$ journalctl -k --since "7 days ago" | grep -i "error|fail|I/O"

# Monitor a specific service in real time
$ journalctl -u nginx.service -f -p warning

Key Takeaways

• Ubuntu uses two parallel logging subsystems: systemd-journald (structured, binary) and rsyslog (traditional text files).
• journalctl provides powerful filtering by unit, priority, time range, PID, and other structured fields.
• Configure journald persistence and size limits in /etc/systemd/journald.conf to prevent journal files from consuming excessive disk space.
• rsyslog routes log messages to categorized text files and can forward logs to remote servers for centralized collection.
• logrotate automatically compresses and removes old log files; always configure rotation for custom application logs.
• Logwatch provides automated daily log summaries delivered by email, highlighting security events and system anomalies.

What’s Next

In the next lesson, we will explore Backup with Bacula and rsnapshot — learning how to implement reliable backup strategies to protect your data and enable disaster recovery.

# 查看所有日誌條目
$ journalctl

# 即時追蹤新條目
$ journalctl -f

# 按 systemd 單元過濾
$ journalctl -u nginx.service

# 按時間範圍過濾
$ journalctl --since "yesterday" --until "today"

# 按優先級過濾
$ journalctl -p err

# 顯示日誌磁碟使用量
$ journalctl --disk-usage

$ sudo nano /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=500M
MaxRetentionSec=30day

# 將日誌轉發到遠端 syslog 伺服器
*.* @@logserver.example.com:514

# 安裝 Logwatch
$ sudo apt install -y logwatch

# 手動執行 Logwatch
$ sudo logwatch --detail High --range today --output stdout

# 設定 Logwatch 預設值
$ sudo nano /etc/logwatch/conf/logwatch.conf

# すべてのジャーナルエントリを表示
$ journalctl

# リアルタイムで新しいエントリをフォロー
$ journalctl -f

# systemd ユニットでフィルタリング
$ journalctl -u nginx.service

# 時間範囲でフィルタリング
$ journalctl --since "yesterday" --until "today"

# 優先度でフィルタリング
$ journalctl -p err

# ジャーナルのディスク使用量を表示
$ journalctl --disk-usage

$ sudo nano /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=500M
MaxRetentionSec=30day

# リモート syslog サーバーにログを転送
*.* @@logserver.example.com:514

# Logwatch をインストール
$ sudo apt install -y logwatch

# Logwatch を手動実行
$ sudo logwatch --detail High --range today --output stdout

# Logwatch のデフォルトを設定
$ sudo nano /etc/logwatch/conf/logwatch.conf