Elastic search cluster , quorum set up - unix1998/technical_notes GitHub Wiki

In an Elasticsearch cluster, quorum and node election processes are primarily handled automatically by Elasticsearch. However, there are certain configurations that you can set to ensure the cluster behaves as expected, especially during network partitions or node failures.

Quorum and Cluster Management

  1. Quorum Setup:

    • Automatically Managed: Elasticsearch automatically handles quorum and the election of the master node. When you set up your cluster, you configure the master-eligible nodes, and Elasticsearch ensures that a quorum (a majority of the master-eligible nodes) is achieved for the cluster to function correctly.
    • Configuration: The primary configuration related to quorum is discovery.zen.minimum_master_nodes (in older versions) or is automatically managed in newer versions (7.x and above) with discovery.type: single-node for single-node clusters.

  2. Cluster Initial Master Nodes:

    • You need to configure the cluster.initial_master_nodes setting with a list of master-eligible nodes during the initial cluster startup. This helps in the initial bootstrapping of the cluster.
    cluster.initial_master_nodes: ["node1", "node2", "node3"]

  3. Majority Rule:

    • Automatic Handling: In Elasticsearch, the master node election follows the majority rule. If the majority of master-eligible nodes are available, they can elect a new master node if the current master node fails.
    • Network Partitions: If a network partition occurs, the majority side (partition with more than half of the master-eligible nodes) will continue to operate as the cluster, while the minority side will not be able to elect a master and will stop accepting indexing or search requests.

Configuration for Majority Handling

In Elasticsearch versions 7.x and above, the configuration for quorum and majority handling is simplified. In older versions (before 7.x), you might need to manually set discovery.zen.minimum_master_nodes.

Elasticsearch 7.x and Above:

  • Auto-managed Quorum: Elasticsearch manages the quorum automatically, and the discovery.zen.minimum_master_nodes setting is deprecated.
  • Configuration Example: 
    cluster.name: my_cluster
node.name: node1
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

Elasticsearch 6.x and Below:

  • Manual Quorum Setting: You need to set discovery.zen.minimum_master_nodes to ensure proper quorum.
  • Setting Example: If you have 3 master-eligible nodes, you should set discovery.zen.minimum_master_nodes to 2 to ensure a majority. 
    cluster.name: my_cluster
node.name: node1
discovery.seed_hosts: ["node1", "node2", "node3"]
discovery.zen.minimum_master_nodes: 2

Example Configuration for a 3-Node Cluster:

For Elasticsearch 7.x and above:

# Common settings for all nodes
cluster.name: my_cluster
network.host: _site_

# Node 1
node.name: node1
node.master: true
node.data: true
path.data: /var/lib/elasticsearch/node1_data
path.logs: /var/log/elasticsearch/node1_logs
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

# Node 2
node.name: node2
node.master: true
node.data: true
path.data: /var/lib/elasticsearch/node2_data
path.logs: /var/log/elasticsearch/node2_logs
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

# Node 3
node.name: node3
node.master: true
node.data: true
path.data: /var/lib/elasticsearch/node3_data
path.logs: /var/log/elasticsearch/node3_logs
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

Handling Node Failures and Network Partitions:

  • Majority Partition: If the majority of master-eligible nodes are available, they will elect a new master and continue operating.
  • Minority Partition: The minority side will not be able to elect a master and will become read-only, effectively preventing split-brain scenarios.

Summary:

  • Automatic Quorum: Elasticsearch handles quorum and master elections automatically, especially in versions 7.x and above.
  • Configuration: Set up cluster.initial_master_nodes for the initial bootstrapping and ensure your discovery.seed_hosts includes all master-eligible nodes.
  • Majority Handling: Elasticsearch follows the majority rule for master elections, ensuring that the cluster remains functional even during node failures or network partitions.