ElasticSearch FailOver and master node setting & AWS elasticSearch Service setting - suguanyu/WechatTinyProgram GitHub Wiki
What is master node:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-node The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node. Any master-eligible node (all nodes by default) may be elected to become the master node by the master election process
' To ensure that your master node is stable and not under pressure, it is a good idea in a bigger cluster to split the roles between dedicated master-eligible nodes and dedicated data nodes. '
Master node re-elect:
To explain, imagine that you have a cluster consisting of two master-eligible nodes. A network failure breaks communication between these two nodes. Each node sees one master-eligible node… itself. With minimum_master_nodes set to the default of 1, this is sufficient to form a cluster. Each node elects itself as the new master (thinking that the other master-eligible node has died) and the result is two clusters, or a split brain. These two nodes will never rejoin until one node is restarted. Any data that has been written to the restarted node will be lost.
Avoiding split brain with minimum_master_nodes
To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting (which defaults to 1) so that each master-eligible node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.
To explain, imagine that you have a cluster consisting of two master-eligible nodes. A network failure breaks communication between these two nodes. Each node sees one master-eligible node… itself. With minimum_master_nodes set to the default of 1, this is sufficient to form a cluster. Each node elects itself as the new master (thinking that the other master-eligible node has died) and the result is two clusters, or a split brain. These two nodes will never rejoin until one node is restarted. Any data that has been written to the restarted node will be lost.
Now imagine that you have a cluster with three master-eligible nodes, and minimum_master_nodes set to 2. If a network split separates one node from the other two nodes, the side with one node cannot see enough master-eligible nodes and will realise that it cannot elect itself as master. The side with two nodes will elect a new master (if needed) and continue functioning correctly. As soon as the network split is resolved, the single node will rejoin the cluster and start serving requests again.
This setting should be set to a quorum of master-eligible nodes:
(master_eligible_nodes / 2) + 1 In other words, if there are three master-eligible nodes, then minimum master nodes should be set to (3 / 2) + 1 or 2:
discovery.zen.minimum_master_nodes: 2
AWS elasticSearch Service setting
For Dedicated master instance count, choose the number of instances for the dedicated master node. We recommend choosing an odd number of instances to avoid potential Amazon ES issues, such as the split brain issue. The default and recommended number is three.