Overview

• What you’ll learn: High-availability clustering concepts, Corosync cluster communication and ring configuration, Pacemaker’s Cluster Resource Manager (CRM), how to define resources using crm and pcs commands, resource types (primitive, clone, group), constraint types (location, colocation, order), and fencing with STONITH devices.
• Prerequisites: DRBD Block Replication (Lesson 65), networking fundamentals, systemd service management
• Estimated reading time: 22 minutes

Introduction

High availability (HA) is the practice of designing systems that continue operating when individual components fail. While backups protect against data loss and DRBD provides real-time data replication, neither can automatically restart services on a surviving node when a server fails. That is the role of a cluster resource manager — and Pacemaker is the leading open-source solution for this task in the Linux ecosystem.

A Pacemaker cluster consists of two or more nodes that communicate through Corosync, a group communication system that provides reliable messaging, membership tracking, and quorum decisions. When Pacemaker detects that a node or service has failed, it automatically migrates resources to a healthy node according to rules defined by the administrator. The entire failover process — detecting failure, fencing the failed node, promoting a replica, starting services, and updating network addresses — happens without human intervention.

In this lesson, we will build a two-node HA cluster on Ubuntu Server, configure Corosync for cluster communication, define Pacemaker resources for a DRBD-backed service, implement fencing with STONITH, and learn the constraint system that governs resource placement and startup ordering.

HA Clustering Concepts

Before diving into configuration, it is important to understand the fundamental concepts that govern cluster behavior:

• Quorum: The minimum number of nodes that must be online for the cluster to make decisions. In a two-node cluster, special quorum policies are required because losing one node means losing 50% of the cluster — normally not a quorum.
• Fencing (STONITH): STONITH stands for “Shoot The Other Node In The Head.” When a node becomes unresponsive, the cluster cannot know whether it has truly failed or is merely unreachable (e.g., network partition). Fencing forcibly powers off or resets the suspect node to guarantee it is not still accessing shared resources. Without fencing, data corruption is possible.
• Resources: The services and components managed by the cluster — filesystems, IP addresses, database servers, web servers, DRBD volumes, etc.
• Constraints: Rules that control where resources run (location), which resources must be on the same node (colocation), and the order in which resources start and stop (order).
• Resource agents: Scripts that Pacemaker uses to start, stop, and monitor resources. They follow the OCF (Open Cluster Framework) standard and are located in /usr/lib/ocf/resource.d/.

Installing Corosync and Pacemaker

We will install the cluster stack on both nodes and establish cluster communication.

# On BOTH nodes — install the cluster stack
$ sudo apt update
$ sudo apt install -y pacemaker corosync pcs resource-agents fence-agents

# Enable and start the PCS daemon (cluster management service)
$ sudo systemctl enable --now pcsd

# Set a password for the cluster admin user (hacluster)
$ sudo passwd hacluster
# Enter a strong password (must be the same on both nodes)

# --- From ONE node only — authenticate and create the cluster ---
$ sudo pcs host auth node1 node2 -u hacluster -p 'YourSecurePassword'
node1: Authorized
node2: Authorized

# Create and start the cluster
$ sudo pcs cluster setup ha-cluster node1 node2
$ sudo pcs cluster start --all
$ sudo pcs cluster enable --all

# Verify cluster status
$ sudo pcs status
Cluster name: ha-cluster
Stack: corosync

Full List of Resources:
  No resources

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

# For a two-node cluster, configure quorum policy
$ sudo pcs property set no-quorum-policy=ignore

# View Corosync ring status
$ sudo corosync-cfgtool -s
Local node ID 1, transport knet
LINK ID 0 addr = 192.168.1.100
  status:
    nodeid: 1 reachable
    nodeid: 2 reachable

Configuring STONITH Fencing

Fencing is not optional in a production cluster — it is a safety requirement. Without fencing, Pacemaker cannot guarantee data integrity after a node failure. The type of fence device depends on your infrastructure.

# List available fence agents
$ sudo pcs stonith list
fence_apc          - APC over telnet/ssh
fence_apc_snmp     - APC over SNMP
fence_idrac        - Dell iDRAC
fence_ipmilan      - IPMI LAN
fence_sbd          - SBD (STONITH Block Device)
fence_virsh        - libvirt (for VMs)
fence_xvm          - XVM (Xen Virtual Machines)

# Example: Configure IPMI-based fencing (common in physical servers)
$ sudo pcs stonith create ipmi-node1 fence_ipmilan 
    pcmk_host_list="node1" 
    ipaddr="192.168.1.200" 
    login="admin" 
    passwd="ipmi-password" 
    lanplus=1 
    op monitor interval=60s

$ sudo pcs stonith create ipmi-node2 fence_ipmilan 
    pcmk_host_list="node2" 
    ipaddr="192.168.1.201" 
    login="admin" 
    passwd="ipmi-password" 
    lanplus=1 
    op monitor interval=60s

# Ensure each node is fenced by the OTHER node's fence device
$ sudo pcs constraint location ipmi-node1 avoids node1=INFINITY
$ sudo pcs constraint location ipmi-node2 avoids node2=INFINITY

# Example: SBD fencing (for environments without IPMI/iLO)
# SBD uses a shared disk as a "poison pill" mechanism
$ sudo pcs stonith create sbd-fence fence_sbd 
    devices="/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBxxxxxxxx" 
    op monitor interval=30s

# Test fencing (WARNING: this will reboot the target node!)
$ sudo pcs stonith fence node2

# Verify fencing is configured
$ sudo pcs stonith status
  ipmi-node1  (stonith:fence_ipmilan): Started node2
  ipmi-node2  (stonith:fence_ipmilan): Started node1

Defining Cluster Resources

Resources are the workloads managed by Pacemaker. We will define a complete HA service stack: a DRBD volume, a filesystem, a virtual IP address, and a web server — all managed as a group with proper ordering.

# Define the DRBD resource (uses the linbit DRBD resource agent)
$ sudo pcs resource create drbd-data ocf:linbit:drbd 
    drbd_resource=r0 
    op monitor interval=15s role=Master 
    op monitor interval=30s role=Slave

# Promote DRBD to a master/slave (promotable) clone
$ sudo pcs resource promotable drbd-data 
    promoted-max=1 promoted-node-max=1 
    clone-max=2 clone-node-max=1 
    notify=true

# Define the filesystem resource (mounts the DRBD device)
$ sudo pcs resource create fs-data ocf:heartbeat:Filesystem 
    device="/dev/drbd0" 
    directory="/data" 
    fstype="ext4" 
    op monitor interval=20s

# Define a virtual IP address for client connections
$ sudo pcs resource create vip-web ocf:heartbeat:IPaddr2 
    ip=192.168.1.50 
    cidr_netmask=24 
    op monitor interval=10s

# Define the application service
$ sudo pcs resource create svc-nginx systemd:nginx 
    op monitor interval=10s 
    op start timeout=30s 
    op stop timeout=30s

# Group the dependent resources (filesystem, VIP, and nginx)
$ sudo pcs resource group add grp-web fs-data vip-web svc-nginx

# --- Define constraints ---

# Colocation: the group must run on the DRBD primary
$ sudo pcs constraint colocation add grp-web with Promoted drbd-data-clone INFINITY

# Order: DRBD must be promoted before the group starts
$ sudo pcs constraint order promote drbd-data-clone then start grp-web

# Verify the full configuration
$ sudo pcs resource status
  * Clone Set: drbd-data-clone [drbd-data] (promotable):
    * Promoted: [ node1 ]
    * Unpromoted: [ node2 ]
  * Resource Group: grp-web:
    * fs-data    (ocf:heartbeat:Filesystem): Started node1
    * vip-web    (ocf:heartbeat:IPaddr2):    Started node1
    * svc-nginx  (systemd:nginx):            Started node1

Resource Types and Constraints Deep Dive

Pacemaker supports several resource types and constraint categories that provide fine-grained control over cluster behavior:

Resource Types

• Primitive: A single instance of a resource. Most resources start as primitives (e.g., a virtual IP, a filesystem mount).
• Clone: A resource that runs on multiple nodes simultaneously. Used for services that should be active everywhere, such as monitoring agents or DRBD in secondary mode.
• Promotable clone (Master/Slave): A clone where one instance can be “promoted” to a special role. DRBD is the classic example — it runs on both nodes, but only one is primary.
• Group: An ordered set of primitives that start sequentially on the same node. If any member fails, the entire group is restarted or migrated.

Constraint Types

• Location constraints: Control which nodes a resource can or cannot run on, with optional scoring to express preference rather than hard requirements.
• Colocation constraints: Require that certain resources run on the same node (positive score) or different nodes (negative score).
• Order constraints: Define the sequence in which resources start and stop. For example, the filesystem must be mounted before the web server starts.

# Location: prefer node1 with a soft score
$ sudo pcs constraint location svc-nginx prefers node1=500

# Location: ban a resource from a specific node
$ sudo pcs constraint location svc-nginx avoids node2=INFINITY

# Colocation: VIP must be on the same node as nginx
$ sudo pcs constraint colocation add vip-web with svc-nginx INFINITY

# Order: filesystem before nginx (mandatory)
$ sudo pcs constraint order fs-data then svc-nginx

# View all constraints
$ sudo pcs constraint show --full

# View cluster configuration as XML
$ sudo pcs cluster cib

Monitoring and Troubleshooting

Effective cluster administration requires monitoring resource status, understanding failover events, and diagnosing problems.

# Real-time cluster status
$ sudo pcs status

# Detailed resource status with failure history
$ sudo pcs resource failcount show

# View the cluster event log
$ sudo pcs status --full

# Check Corosync membership
$ sudo corosync-cmapctl | grep members

# View Pacemaker logs
$ journalctl -u pacemaker -f

# Simulate a failover (move resources to the other node)
$ sudo pcs resource move grp-web node2

# Clear resource constraints created by move
$ sudo pcs resource clear grp-web

# Put a node into standby (gracefully drain resources)
$ sudo pcs node standby node1

# Bring a node back from standby
$ sudo pcs node unstandby node1

# Clean up failed resource state
$ sudo pcs resource cleanup svc-nginx

Key Takeaways

• Pacemaker provides automated resource management and failover for HA clusters, working with Corosync for cluster communication, membership, and quorum.
• STONITH fencing is mandatory in production clusters — it guarantees that a failed node cannot corrupt shared resources by forcibly powering it off before migrating services.
• Resources are defined as primitives, clones, promotable clones, or groups; constraints (location, colocation, order) control where and in what sequence they run.
• The pcs command is the primary administration tool for modern Pacemaker clusters, handling cluster setup, resource definition, constraint management, and troubleshooting.
• A complete HA stack typically layers Corosync (messaging) + Pacemaker (decisions) + DRBD (data) + STONITH (safety) to deliver zero-downtime failover.

What’s Next

Congratulations! You have completed the Monitoring & High Availability module — and the entire Linux Engineering track! You now possess a comprehensive skill set spanning system fundamentals, networking, security, storage, automation, and high availability. Continue honing these skills through hands-on practice in lab environments, contribute to open-source projects, and pursue professional certifications to validate your expertise.

# 在兩個節點上安裝叢集堆疊
$ sudo apt install -y pacemaker corosync pcs resource-agents fence-agents

# 啟用 PCS 守護程序
$ sudo systemctl enable --now pcsd

# 設定叢集管理使用者密碼
$ sudo passwd hacluster

# 認證並建立叢集（僅從一個節點執行）
$ sudo pcs host auth node1 node2 -u hacluster -p 'YourSecurePassword'
$ sudo pcs cluster setup ha-cluster node1 node2
$ sudo pcs cluster start --all
$ sudo pcs cluster enable --all

# 雙節點叢集仲裁策略
$ sudo pcs property set no-quorum-policy=ignore

# 驗證叢集狀態
$ sudo pcs status

# 列出可用的隔離代理
$ sudo pcs stonith list

# 設定 IPMI 隔離（實體伺服器常用）
$ sudo pcs stonith create ipmi-node1 fence_ipmilan 
    pcmk_host_list="node1" 
    ipaddr="192.168.1.200" 
    login="admin" passwd="ipmi-password" lanplus=1

$ sudo pcs stonith create ipmi-node2 fence_ipmilan 
    pcmk_host_list="node2" 
    ipaddr="192.168.1.201" 
    login="admin" passwd="ipmi-password" lanplus=1

# 確保每個節點由另一個節點的隔離裝置隔離
$ sudo pcs constraint location ipmi-node1 avoids node1=INFINITY
$ sudo pcs constraint location ipmi-node2 avoids node2=INFINITY

# 定義 DRBD 資源
$ sudo pcs resource create drbd-data ocf:linbit:drbd 
    drbd_resource=r0 
    op monitor interval=15s role=Master 
    op monitor interval=30s role=Slave

# 提升 DRBD 為可提升的克隆
$ sudo pcs resource promotable drbd-data 
    promoted-max=1 promoted-node-max=1 
    clone-max=2 clone-node-max=1 notify=true

# 定義檔案系統資源
$ sudo pcs resource create fs-data ocf:heartbeat:Filesystem 
    device="/dev/drbd0" directory="/data" fstype="ext4"

# 定義虛擬 IP 位址
$ sudo pcs resource create vip-web ocf:heartbeat:IPaddr2 
    ip=192.168.1.50 cidr_netmask=24

# 定義應用程式服務
$ sudo pcs resource create svc-nginx systemd:nginx

# 群組化相依資源
$ sudo pcs resource group add grp-web fs-data vip-web svc-nginx

# 共置約束：群組必須在 DRBD 主節點上運行
$ sudo pcs constraint colocation add grp-web with Promoted drbd-data-clone INFINITY

# 順序約束：DRBD 必須先提升，然後群組才啟動
$ sudo pcs constraint order promote drbd-data-clone then start grp-web

# 即時叢集狀態
$ sudo pcs status

# 模擬容錯移轉
$ sudo pcs resource move grp-web node2

# 清除移動產生的約束
$ sudo pcs resource clear grp-web

# 將節點置於待機模式
$ sudo pcs node standby node1
$ sudo pcs node unstandby node1

# 清理失敗的資源狀態
$ sudo pcs resource cleanup svc-nginx

# 両ノードにクラスタースタックをインストール
$ sudo apt install -y pacemaker corosync pcs resource-agents fence-agents

# PCS デーモンを有効化して起動
$ sudo systemctl enable --now pcsd

# クラスター管理ユーザーのパスワードを設定
$ sudo passwd hacluster

# 認証してクラスターを作成（1つのノードからのみ）
$ sudo pcs host auth node1 node2 -u hacluster -p 'YourSecurePassword'
$ sudo pcs cluster setup ha-cluster node1 node2
$ sudo pcs cluster start --all
$ sudo pcs cluster enable --all

# 2ノードクラスターのクォーラムポリシー
$ sudo pcs property set no-quorum-policy=ignore

# クラスターステータスを確認
$ sudo pcs status

# 利用可能なフェンスエージェントを一覧表示
$ sudo pcs stonith list

# IPMI ベースのフェンシングを設定（物理サーバーで一般的）
$ sudo pcs stonith create ipmi-node1 fence_ipmilan 
    pcmk_host_list="node1" 
    ipaddr="192.168.1.200" 
    login="admin" passwd="ipmi-password" lanplus=1

$ sudo pcs stonith create ipmi-node2 fence_ipmilan 
    pcmk_host_list="node2" 
    ipaddr="192.168.1.201" 
    login="admin" passwd="ipmi-password" lanplus=1

# 各ノードが他方のノードのフェンスデバイスでフェンシングされることを確保
$ sudo pcs constraint location ipmi-node1 avoids node1=INFINITY
$ sudo pcs constraint location ipmi-node2 avoids node2=INFINITY

# DRBD リソースを定義
$ sudo pcs resource create drbd-data ocf:linbit:drbd 
    drbd_resource=r0 
    op monitor interval=15s role=Master 
    op monitor interval=30s role=Slave

# DRBD をプロモータブルクローンに昇格
$ sudo pcs resource promotable drbd-data 
    promoted-max=1 promoted-node-max=1 
    clone-max=2 clone-node-max=1 notify=true

# ファイルシステムリソースを定義
$ sudo pcs resource create fs-data ocf:heartbeat:Filesystem 
    device="/dev/drbd0" directory="/data" fstype="ext4"

# 仮想 IP アドレスを定義
$ sudo pcs resource create vip-web ocf:heartbeat:IPaddr2 
    ip=192.168.1.50 cidr_netmask=24

# アプリケーションサービスを定義
$ sudo pcs resource create svc-nginx systemd:nginx

# 依存リソースをグループ化
$ sudo pcs resource group add grp-web fs-data vip-web svc-nginx

# コロケーション制約：グループは DRBD プライマリで実行
$ sudo pcs constraint colocation add grp-web with Promoted drbd-data-clone INFINITY

# 順序制約：DRBD が昇格してからグループが開始
$ sudo pcs constraint order promote drbd-data-clone then start grp-web

# リアルタイムクラスターステータス
$ sudo pcs status

# フェイルオーバーをシミュレート
$ sudo pcs resource move grp-web node2

# move で作成された制約をクリア
$ sudo pcs resource clear grp-web

# ノードをスタンバイモードにする
$ sudo pcs node standby node1
$ sudo pcs node unstandby node1

# 失敗したリソース状態をクリーンアップ
$ sudo pcs resource cleanup svc-nginx