Setup ETCD Cluster - hqzhang/cloudtestbed GitHub Wiki

Etcd is a distributed key/value store and also provide watch feature for value changing. Etcd cluster can make this kind database more reliable and safty. cofiguration for etcd cluster is shown as following: Three nodes is recommand minimum number of cluter i)restart testing name=infra0 IP=192.168.56.103 //cluster should list all member of cluster(including itself) cluster="infra0=http://192.168.56.103:2380,http://192.168.56.104:2380,http://192.168.56.105:2380" //cluster-state must be NEW, keey data-dir data state=new /usr/bin/etcd2 --name ${name}
--data-dir /var/lib/etcd
--listen-client-urls http://${IP}:2379,http://127.0.0.1:2379
--advertise-client-urls http://${IP}:2379
--listen-peer-urls http://${IP}:2380
--initial-advertise-peer-urls http://${IP}:2380
--initial-cluster $cluster
--initial-cluster-token my-etcd-token
--initial-cluster-state $state

ii) add the node testing: name=infra0 IP=192.168.56.103 //cluster should include all member of cluster(including itself) cluster="infra0=http://192.168.56.103:2380,http://192.168.56.104:2380,http://192.168.56.105:2380" //cluster-state should be "existing" and removing data-dir data state=new /usr/bin/etcd2 --name ${name}
--data-dir /var/lib/etcd
--listen-client-urls http://${IP}:2379,http://127.0.0.1:2379
--advertise-client-urls http://${IP}:2379
--listen-peer-urls http://${IP}:2380
--initial-advertise-peer-urls http://${IP}:2380
--initial-cluster $cluster
--initial-cluster-token my-etcd-token
--initial-cluster-state $state

#Static config with one node and then increasing

This method shows that first starting one node cluster and then add more node into cluster assuming we have three nodes: 192.168.56.103/192.168.56.104/192.168.56.105, the step is shown as following:

1)on node 192.168.56.103. name=infra0 IP=192.168.56.103 cluster="infra0=http://$IP:2380" state=new /usr/bin/etcd2 --name ${name}
--data-dir /var/lib/etcd
--listen-client-urls http://${IP}:2379,http://127.0.0.1:2379
--advertise-client-urls http://${IP}:2379
--listen-peer-urls http://${IP}:2380
--initial-advertise-peer-urls http://${IP}:2380
--initial-cluster $cluster
--initial-cluster-token my-etcd-token
--initial-cluster-state $state 2) etcdctl member list 4a84c8889f5d7a35: name=infra0 peerURLs=http://192.168.56.103:2380 clientURLs=http://192.168.56.103:2379 isLeader=true

3. on node 192.168.56.103 ,add another node 192.168.56.104 etcdctl member add infra1 http://192.168.56.104:2380 Added member named infra1 with ID 2ab43b37d3551726 to cluster

ETCD_NAME="infra1" ETCD_INITIAL_CLUSTER="infra1=http://192.168.56.104:2380,infra0=http://192.168.56.103:2380" ETCD_INITIAL_CLUSTER_STATE="existing"

4. etcdctl member list 2ab43b37d3551726[unstarted]: peerURLs=http://192.168.56.104:2380 4a84c8889f5d7a35: name=infra0 peerURLs=http://192.168.56.103:2380 clientURLs=http://192.168.56.103:2379 isLeader=true

5. start it on the node 192.168.56.104 name=infra1 IP=192.168.56.104 cluster="infra0=http://192.168.56.103,infra1=http://$IP:2380" state="existing" /usr/bin/etcd2 --name ${name}
   --data-dir /var/lib/etcd
   --listen-client-urls http://${IP}:2379,http://127.0.0.1:2379
   --advertise-client-urls http://${IP}:2379
   --listen-peer-urls http://${IP}:2380
   --initial-advertise-peer-urls http://${IP}:2380
   --initial-cluster $cluster
   --initial-cluster-token my-etcd-token
   --initial-cluster-state $state

6. etcdctl member list 4a84c8889f5d7a35: name=infra0 peerURLs=http://192.168.56.103:2380 clientURLs=http://192.168.56.103:2379 isLeader=true b5c42f95d580844d: name=infra1 peerURLs=http://192.168.56.104:2380 clientURLs=http://192.168.56.104:2379 isLeader=false

#Conclusion:

1. Choose static configuration, even initial size=1 (discovery can do the same thing, there is no difference if size=1)
2. Three node is minimum size of cluster for disaster recovery( one totally remove and then can be recoveried completely)
3. Any size of cluster can do restart.