Overview

• What you’ll learn: Block-level replication concepts and how they differ from file-level replication, DRBD resource configuration including disk, network, and syncer settings, primary/secondary role management, split-brain detection and recovery, key drbdadm commands, and performance tuning with replication protocols A, B, and C.
• Prerequisites: Backup with Bacula and rsnapshot (Lesson 64), LVM basics, networking fundamentals
• Estimated reading time: 22 minutes

Introduction

Traditional backups protect against data loss but introduce a recovery gap — the time between the last backup and the failure event. In mission-critical environments where even minutes of data loss are unacceptable, you need continuous data protection. DRBD (Distributed Replicated Block Device) provides exactly this by replicating block devices in real time between two or more servers over a network connection.

Think of DRBD as a network RAID-1 mirror. Just as a local RAID-1 array mirrors writes across two physical disks in the same machine, DRBD mirrors writes across two disks on separate machines connected by a network. When the primary server writes a block to its local disk, DRBD simultaneously transmits that block to the secondary server, keeping both copies synchronized at the block level.

DRBD operates in the Linux kernel as a virtual block device driver. Applications and filesystems interact with the DRBD device (/dev/drbd0) exactly as they would with any other block device. The replication is transparent — no application changes are required. This makes DRBD an ideal foundation for high-availability clusters, where it is commonly paired with Pacemaker and Corosync to provide automatic failover.

Block-Level vs. File-Level Replication

Understanding why block-level replication matters requires comparing it with file-level alternatives:

• File-level replication (e.g., rsync, lsyncd) copies entire files when changes are detected. For a 10 GB database file where one record changes, the entire file may need to be transferred or at least checksummed. File-level tools also struggle with open files and consistency.
• Block-level replication (DRBD) operates below the filesystem. When an application writes even a single byte, only the affected disk blocks (typically 4 KB) are replicated. This is dramatically more efficient for databases, virtual machines, and any workload with large files that change incrementally.

Block-level replication also guarantees write-order consistency. Writes arrive at the replica in the same order they were issued on the primary, ensuring that the secondary always has a crash-consistent copy of the data.

DRBD Replication Protocols

DRBD offers three replication protocols that define when a write operation is considered complete. Choosing the right protocol is a critical trade-off between data safety and write latency:

• Protocol A (asynchronous): A write is considered complete as soon as it reaches the local disk and the network send buffer. The primary does not wait for the secondary to acknowledge receipt. This offers the best write performance but risks data loss if the primary fails before the secondary receives the data. Suitable for long-distance replication where latency makes synchronous replication impractical.
• Protocol B (semi-synchronous / memory-synchronous): A write is considered complete when it reaches the local disk and the secondary has received the data in memory (but not yet flushed to disk). This protects against primary failure but not simultaneous dual-node failure.
• Protocol C (synchronous): A write is considered complete only when it has been written to both the local disk and the remote disk. This provides the strongest data guarantee — zero data loss on primary failure — but adds network round-trip latency to every write operation. This is the recommended protocol for HA clusters on a local network.

Installing and Configuring DRBD

We will set up a DRBD resource between two Ubuntu servers to replicate a partition in real time.

# On BOTH nodes — install DRBD
$ sudo apt update
$ sudo apt install -y drbd-utils

# Verify kernel module is available
$ sudo modprobe drbd
$ lsmod | grep drbd
drbd                  409600  0

# Prepare a partition or LVM volume for DRBD on BOTH nodes
# Using LVM (recommended):
$ sudo lvcreate -n drbd-data -L 50G vg0

# --- DRBD resource configuration ---
# Create the resource file on BOTH nodes (must be identical)
$ sudo tee /etc/drbd.d/r0.res > /dev/null <<'EOF'
resource r0 {
  protocol C;

  net {
    # Verify data integrity on the wire
    verify-alg sha256;
    # Timeout and keepalive settings
    timeout 60;         # 6 seconds (unit: 0.1s)
    connect-int 10;     # reconnect interval in seconds
    ping-int 10;        # keepalive ping interval
    ping-timeout 5;     # ping response timeout
  }

  disk {
    # Resync rate limit (adjust based on network capacity)
    resync-rate 100M;
    # Enable online verification
    on-io-error detach;
  }

  on node1 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.100:7789;
    meta-disk internal;
  }

  on node2 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.101:7789;
    meta-disk internal;
  }
}
EOF

# Initialize the DRBD metadata on BOTH nodes
$ sudo drbdadm create-md r0
initializing activity log
initializing bitmap (1600 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

# Start the DRBD resource on BOTH nodes
$ sudo drbdadm up r0

# On the DESIGNATED PRIMARY node — force initial sync
$ sudo drbdadm primary --force r0

# Monitor the initial synchronization
$ watch cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent
    [====>...............] sync'ed: 28.4% (36224/50000)M

Day-to-Day DRBD Operations

Once the initial synchronization is complete, the DRBD resource is ready for use. The primary node can create a filesystem on the device and mount it.

# Create a filesystem on the DRBD device (PRIMARY only)
$ sudo mkfs.ext4 /dev/drbd0
$ sudo mkdir -p /data
$ sudo mount /dev/drbd0 /data

# Check DRBD status
$ sudo drbdadm status r0
r0 role:Primary
  disk:UpToDate
  node2 role:Secondary
    peer-disk:UpToDate

# Detailed connection info
$ sudo drbdsetup status r0 --verbose --statistics

# --- Manual failover procedure ---
# On the PRIMARY node — demote and unmount
$ sudo umount /data
$ sudo drbdadm secondary r0

# On the NEW PRIMARY node — promote and mount
$ sudo drbdadm primary r0
$ sudo mount /dev/drbd0 /data

# Verify data integrity with online verification
$ sudo drbdadm verify r0
# Check results
$ sudo drbdadm status r0
# Look for "out-of-sync" count — should be 0

# Temporarily disconnect for maintenance
$ sudo drbdadm disconnect r0
# Reconnect when maintenance is complete
$ sudo drbdadm connect r0

Split-Brain Detection and Recovery

Split-brain is the most dangerous failure scenario in any replicated system. It occurs when both nodes become primary simultaneously — typically because the network link between them fails while both nodes continue operating. Each node accepts writes independently, creating divergent datasets that cannot be automatically merged.

DRBD detects split-brain upon reconnection and refuses to synchronize until an administrator resolves the conflict. Automated split-brain recovery policies can be configured, but they inherently discard data from one side.

# Configure automatic split-brain recovery policies
# Add to the net { } section of the resource definition:
net {
  # If one node had no changes, auto-resolve
  after-sb-0pri discard-zero-changes;
  # If one node was primary, keep its data
  after-sb-1pri discard-secondary;
  # If both were primary, disconnect (manual intervention required)
  after-sb-2pri disconnect;
}

# --- Manual split-brain recovery ---
# On the VICTIM node (whose data you want to DISCARD):
$ sudo drbdadm secondary r0
$ sudo drbdadm disconnect r0
$ sudo drbdadm -- --discard-my-data connect r0

# On the SURVIVING node (whose data you want to KEEP):
$ sudo drbdadm connect r0

# The victim node will resync from the survivor
$ sudo drbdadm status r0

Performance Tuning

DRBD performance depends on network bandwidth, latency, and disk I/O speed. Several parameters can be tuned to optimize throughput.

# Key performance parameters in the resource definition:

resource r0 {
  net {
    # Maximum number of write requests in flight
    max-buffers 8000;
    max-epoch-size 8000;

    # Sndbuf/rcvbuf size (0 = auto-tune)
    sndbuf-size 0;
    rcvbuf-size 0;
  }

  disk {
    # Resync rate — maximum bandwidth for background sync
    resync-rate 200M;

    # Use checksum-based resync for efficiency
    c-plan-ahead 20;
    c-max-rate 500M;
    c-fill-target 24M;

    # Align I/O requests for better SSD performance
    disk-barrier no;
    disk-flushes no;     # Only if battery-backed cache
  }
}

# Monitor DRBD performance metrics
$ sudo drbdsetup status r0 --verbose --statistics
# Key metrics to watch:
#   - send/receive throughput (KB/s)
#   - pending/unacked writes
#   - out-of-sync blocks (should be 0 in steady state)

# Benchmark write performance
$ sudo dd if=/dev/zero of=/data/testfile bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.23 s, 205 MB/s

Key Takeaways

• DRBD provides real-time block-level replication between servers, functioning as a network RAID-1 mirror that is transparent to applications and filesystems above it.
• Protocol C (synchronous) guarantees zero data loss on primary failure and is recommended for HA clusters on local networks; Protocols A and B trade data safety for lower write latency.
• Split-brain is the most critical failure scenario; it occurs when both nodes accept writes independently, and recovery always involves discarding one side’s changes.
• The drbdadm command is the primary administration tool — use it for promotion, demotion, connection management, status checks, and online verification.
• Performance tuning involves matching DRBD buffer sizes, resync rates, and I/O barriers to your network bandwidth, latency, and storage hardware capabilities.

What’s Next

In the next lesson, we will explore Pacemaker HA Clustering — learning how to build on DRBD with Corosync and Pacemaker to create a fully automated high-availability cluster with resource management, fencing, and automatic failover.

# 在兩個節點上安裝 DRBD
$ sudo apt install -y drbd-utils

# 驗證核心模組可用
$ sudo modprobe drbd

# 為 DRBD 準備 LVM 卷
$ sudo lvcreate -n drbd-data -L 50G vg0

# 建立資源設定檔（兩個節點必須相同）
resource r0 {
  protocol C;
  on node1 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.100:7789;
    meta-disk internal;
  }
  on node2 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.101:7789;
    meta-disk internal;
  }
}

# 初始化中繼資料並啟動
$ sudo drbdadm create-md r0
$ sudo drbdadm up r0
$ sudo drbdadm primary --force r0   # 僅在指定的主節點上

# 在 DRBD 裝置上建立檔案系統（僅主節點）
$ sudo mkfs.ext4 /dev/drbd0
$ sudo mount /dev/drbd0 /data

# 檢查狀態
$ sudo drbdadm status r0

# 手動容錯移轉
# 在主節點上：
$ sudo umount /data
$ sudo drbdadm secondary r0

# 在新主節點上：
$ sudo drbdadm primary r0
$ sudo mount /dev/drbd0 /data

# 線上驗證資料完整性
$ sudo drbdadm verify r0

# 設定自動腦裂復原策略
net {
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri disconnect;
}

# 手動腦裂復原
# 在受害節點上（要捨棄其資料的節點）：
$ sudo drbdadm secondary r0
$ sudo drbdadm disconnect r0
$ sudo drbdadm -- --discard-my-data connect r0

# 在存活節點上：
$ sudo drbdadm connect r0

# 両ノードに DRBD をインストール
$ sudo apt install -y drbd-utils

# カーネルモジュールが利用可能であることを確認
$ sudo modprobe drbd

# DRBD 用の LVM ボリュームを準備
$ sudo lvcreate -n drbd-data -L 50G vg0

# リソース設定ファイルを作成（両ノードで同一）
resource r0 {
  protocol C;
  on node1 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.100:7789;
    meta-disk internal;
  }
  on node2 {
    device    /dev/drbd0;
    disk      /dev/vg0/drbd-data;
    address   192.168.1.101:7789;
    meta-disk internal;
  }
}

# メタデータを初期化して起動
$ sudo drbdadm create-md r0
$ sudo drbdadm up r0
$ sudo drbdadm primary --force r0   # 指定したプライマリノードでのみ

# DRBD デバイスにファイルシステムを作成（プライマリのみ）
$ sudo mkfs.ext4 /dev/drbd0
$ sudo mount /dev/drbd0 /data

# ステータスを確認
$ sudo drbdadm status r0

# 手動フェイルオーバー
# プライマリノードで：
$ sudo umount /data
$ sudo drbdadm secondary r0

# 新しいプライマリノードで：
$ sudo drbdadm primary r0
$ sudo mount /dev/drbd0 /data

# オンラインでデータ整合性を検証
$ sudo drbdadm verify r0

# 自動スプリットブレイン復旧ポリシーの設定
net {
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri disconnect;
}

# 手動スプリットブレイン復旧
# 被害ノードで（データを破棄するノード）：
$ sudo drbdadm secondary r0
$ sudo drbdadm disconnect r0
$ sudo drbdadm -- --discard-my-data connect r0

# 存続ノードで：
$ sudo drbdadm connect r0