U1.44 Ubuntu Quick Start (QS): Etcd cluster on premises - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki

Reading

We start with

Prepare u2004m01 u2004m02 u2004m03

Download and install binaries

Creating etcd folders

Static bootstrapping an etcd cluster without TLS

Static bootstrapping an etcd cluster with TLS Automatic certificates

SelfSigned certificates

Prometheus

Grafana

Reading

read the article Bootstrapping the etcd Cluster
read the article Install pre-built binaries
read the article Clustering Guide

We start with

Pre-installed DHCP in the virtual environment (for example, a hardware implementation of a DHCP server in a modem)
Go to the page Ubuntu 20.04.3 LTS (Focal Fossa)
Download ubuntu-20.04.3-live-server-amd64.iso
Deploy three virtual machines with default settings (i.e. openssh is ON)
- u2004s01 192.168.100.11
- u2004s02 192.168.100.9
- u2004s03 192.168.100.12

Prepare u2004m01 u2004m02 u2004m03

Step 1:
- run the commands to set the password for root-user

sudo -i
passwd

Step 2:
- with sudo nano /etc/ssh/sshd_config modify the file
  - set: PermitRootLogin yes
Step 3:
- run the command

sudo service ssh restart

Download and install binaries

for each u2004s01, u2004s02, u2004s03 run the commands

wget -q --show-progress --https-only --timestamping "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz"
ls -l
tar -xvf  etcd-v3.5.0-linux-amd64.tar.gz
sudo mv etcd-v3.5.0-linux-amd64/etcd/* /usr/local/bin/
etcd --version

Click to show the responce

yury@u2004s01:~$ wget -q --show-progress --https-only --timestamping "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz"
etcd-v3.5.0-linux-amd64.tar.gz                100%[===============================================================================================>]  18.49M  10.3MB/s    in 1.8s

yury@u2004s01:~$ ls -l
total 18936
-rw-rw-r-- 1 yury yury 19389988 Dec  6 19:43 etcd-v3.5.0-linux-amd64.tar.gz

yury@u2004s01:~$ tar -xvf  etcd-v3.5.0-linux-amd64.tar.gz
etcd-v3.5.0-linux-amd64/
etcd-v3.5.0-linux-amd64/Documentation/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/README.md
etcd-v3.5.0-linux-amd64/README-etcdutl.md
etcd-v3.5.0-linux-amd64/READMEv2-etcdctl.md
etcd-v3.5.0-linux-amd64/README-etcdctl.md
etcd-v3.5.0-linux-amd64/README.md
etcd-v3.5.0-linux-amd64/etcdutl
etcd-v3.5.0-linux-amd64/etcdctl
etcd-v3.5.0-linux-amd64/etcd

yury@u2004s01:~$ sudo mv etcd-v3.5.0-linux-amd64/etcd* /usr/local/bin/

yury@u2004s01:~$ etcd --version
etcd Version: 3.5.0
Git SHA: 946a5a6f2
Go Version: go1.16.3
Go OS/Arch: linux/amd64

Creating etcd folders

for each u2004s01, u2004s02, u2004s03 run the commands

sudo mkdir -p /etc/etcd /var/lib/etcd
sudo chmod 700 /var/lib/etcd

Static bootstrapping an etcd cluster without TLS

for each u2004s01, u2004s02, u2004s03 run the command

sudo nano /etc/systemd/system/etcd.service

insert the following content for u2004s01

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s01 \
  --initial-advertise-peer-urls http://192.168.100.11:2380 \
  --listen-peer-urls http://192.168.100.11:2380 \
  --listen-client-urls http://192.168.100.11:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.100.11:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=http://192.168.100.11:2380,u2004s02=http://192.168.100.9:2380,u2004s03=http://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s02

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s02 \
  --initial-advertise-peer-urls http://192.168.100.9:2380 \
  --listen-peer-urls http://192.168.100.9:2380 \
  --listen-client-urls http://192.168.100.9:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.100.9:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=http://192.168.100.11:2380,u2004s02=http://192.168.100.9:2380,u2004s03=http://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd 
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s03

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s03 \
  --initial-advertise-peer-urls http://192.168.100.12:2380 \
  --listen-peer-urls http://192.168.100.12:2380 \
  --listen-client-urls http://192.168.100.12:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.100.12:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=http://192.168.100.11:2380,u2004s02=http://192.168.100.9:2380,u2004s03=http://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd 
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
sudo etcdctl member list

yury@u2004s03:~$ sudo etcdctl member list
3f11c9d611f6df65, started, u2004s01, http://192.168.100.11:2380, http://192.168.100.11:2379, false
60b4b1ea4e5cf084, started, u2004s03, http://192.168.100.12:2380, http://192.168.100.12:2379, false
9ffa647521f6cfba, started, u2004s02, http://192.168.100.9:2380, http://192.168.100.9:2379, false

Test

for u2004s03 run the command

yury@u2004s03:~$ sudo etcdctl put foo "Hello World!"
OK

for u2004s01 run the command

yury@u2004s01:~$ etcdctl  get foo
foo
Hello World!
yury@u2004s01:~$ etcdctl del foo
1

Destroy the cluster

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl stop etcd
sudo systemctl disable etcd
sudo rm /etc/systemd/system/etcd.service
sudo systemctl daemon-reload

Static bootstrapping an etcd cluster with TLS Automatic certificates

Note: If the cluster needs encrypted communication but does not require authenticated connections, etcd can be configured to automatically generate its keys. On initialization, each member creates its own set of keys based on its advertised IP addresses and hosts.
for each u2004s01, u2004s02, u2004s03 run the command

sudo nano /etc/systemd/system/etcd.service

insert the following content for u2004s01

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s01 \
  --initial-advertise-peer-urls https://192.168.100.11:2380 \
  --listen-peer-urls https://192.168.100.11:2380 \
  --listen-client-urls https://192.168.100.11:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.11:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --auto-tls \
  --peer-auto-tls
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s02

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s02 \
  --initial-advertise-peer-urls https://192.168.100.9:2380 \
  --listen-peer-urls https://192.168.100.9:2380 \
  --listen-client-urls https://192.168.100.9:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.9:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --auto-tls \
  --peer-auto-tls
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s03

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s03 \
  --initial-advertise-peer-urls https://192.168.100.12:2380 \
  --listen-peer-urls https://192.168.100.12:2380 \
  --listen-client-urls https://192.168.100.12:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.12:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --auto-tls \
  --peer-auto-tls
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd


yury@u2004s03:~$ sudo ls -l /var/lib/etcd/fixtures/client/
total 8
-rw-r--r-- 1 root root 733 Dec 16 18:14 cert.pem
-rw------- 1 root root 365 Dec 16 18:14 key.pem


yury@u2004s03:~$ sudo etcdctl member list --insecure-skip-tls-verify  --cert /var/lib/etcd/fixtures/client/cert.pem --key /var/lib/etcd/fixtures/client/key.pem --endpoints https://127.0.0.1:2379
f592fd9ac4ffa37, started, u2004s01, https://192.168.100.11:2380, https://192.168.100.11:2379, false
2c1390a86cc037db, started, u2004s03, https://192.168.100.12:2380, https://192.168.100.12:2379, false
49d66ef58fa104e5, started, u2004s02, https://192.168.100.9:2380, https://192.168.100.9:2379, false

next command is correct as well

sudo etcdctl member list --insecure-skip-tls-verify --cert /var/lib/etcd/fixtures/client/cert.pem --key /var/lib/etcd/fixtures/client/key.pem

Destroy the cluster

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl stop etcd
sudo systemctl disable etcd
sudo rm /etc/systemd/system/etcd.service

yury@u2004s03:~$ sudo ls -l /var/lib/etcd/
drwx------ 4 root root 4096 Dec 16 18:14 fixtures
drwx------ 4 root root 4096 Dec 16 18:14 member

sudo rm -r /var/lib/etcd/fixtures
sudo rm -r /var/lib/etcd/member
sudo systemctl daemon-reload

SelfSigned certificates

Note: A cluster using self-signed certificates both encrypts traffic and authenticates its connections.

Generating Self-signed certificates

read the article tls-setup

Download "tls-setup" folder

for u2004s01 run the conmmands

mkdir tlssetup
cd tlssetup
git init
git remote add origin https://github.com/etcd-io/etcd/
git sparse-checkout init
git sparse-checkout set tls-setup
git pull origin main 
cd hack/tls-setup

Install "make" utility

for u2004s01 run the conmmands

sudo apt-get update -y
sudo apt-get install -y make

Install "golang-cfssl" utility

for u2004s01 run the conmmands

sudo apt-get update -y
sudo apt install golang-cfssl

Modify config/req-csr.json

for u2004s01 run the conmmands

yury@u2004s01:~/tlssetup/hack/tls-setup$ nano   config/req-csr.json

insert the following content

{
  "CN": "etcd",
  "hosts": [
    "localhost",
    "127.0.0.1",
    "192.168.100.11",
    "192.168.100.9",
    "192.168.100.12"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "autogenerated",
      "OU": "etcd cluster",
      "L": "the internet"
    }
  ]
}

Generate certs

for u2004s01 run the conmmands

export infra0={192.168.100.11}
export infra1={192.168.100.9}
export infra2={192.168.100.12}
make
ls -l certs/

Click to show the responce

yury@u2004s01:~/tlssetup/hack/tls-setup$ export infra0={192.168.100.11}
yury@u2004s01:~/tlssetup/hack/tls-setup$ export infra1={192.168.100.9}
yury@u2004s01:~/tlssetup/hack/tls-setup$ export infra2={192.168.100.12}
yury@u2004s01:~/tlssetup/hack/tls-setup$ make
mkdir -p certs
2021/12/17 15:23:56 [INFO] generating a new CA key and certificate from CSR
2021/12/17 15:23:56 [INFO] generate received request
2021/12/17 15:23:56 [INFO] received CSR
2021/12/17 15:23:56 [INFO] generating key: rsa-2048
2021/12/17 15:23:56 [INFO] encoded CSR
2021/12/17 15:23:56 [INFO] signed certificate with serial number 318013364448825257411727924924727436382500973190
2021/12/17 15:23:56 [INFO] generate received request
2021/12/17 15:23:56 [INFO] received CSR
2021/12/17 15:23:56 [INFO] generating key: rsa-2048
2021/12/17 15:23:57 [INFO] encoded CSR
2021/12/17 15:23:57 [INFO] signed certificate with serial number 616692823799396961459104761389647158413682902471
2021/12/17 15:23:57 [INFO] generate received request
2021/12/17 15:23:57 [INFO] received CSR
2021/12/17 15:23:57 [INFO] generating key: rsa-2048
2021/12/17 15:23:57 [INFO] encoded CSR
2021/12/17 15:23:57 [INFO] signed certificate with serial number 33290944541938277959621438482007664421628312569
2021/12/17 15:23:57 [INFO] generate received request
2021/12/17 15:23:57 [INFO] received CSR
2021/12/17 15:23:57 [INFO] generating key: rsa-2048
2021/12/17 15:23:57 [INFO] encoded CSR
2021/12/17 15:23:57 [INFO] signed certificate with serial number 141549358363476941144917361864327161588449726647
2021/12/17 15:23:57 [INFO] generate received request
2021/12/17 15:23:57 [INFO] received CSR
2021/12/17 15:23:57 [INFO] generating key: rsa-2048
2021/12/17 15:23:57 [INFO] encoded CSR
2021/12/17 15:23:57 [INFO] signed certificate with serial number 415930075839779861707935550197121049972799255116
2021/12/17 15:23:57 [INFO] generate received request
2021/12/17 15:23:57 [INFO] received CSR
2021/12/17 15:23:57 [INFO] generating key: rsa-2048
2021/12/17 15:23:57 [INFO] encoded CSR
2021/12/17 15:23:57 [INFO] signed certificate with serial number 565420498652855834328343139374977789553654416371
2021/12/17 15:23:57 [INFO] generate received request
2021/12/17 15:23:57 [INFO] received CSR
2021/12/17 15:23:57 [INFO] generating key: rsa-2048
2021/12/17 15:23:58 [INFO] encoded CSR
2021/12/17 15:23:58 [INFO] signed certificate with serial number 68634681476893248279406333674947451000898536333
yury@u2004s01:~/tlssetup/hack/tls-setup$ ls -l certs/
total 84
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 {192.168.100.11}.csr
-rw------- 1 yury yury 1675 Dec 17 15:23 {192.168.100.11}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 {192.168.100.11}.pem
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 {192.168.100.12}.csr
-rw------- 1 yury yury 1679 Dec 17 15:23 {192.168.100.12}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 {192.168.100.12}.pem
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 {192.168.100.9}.csr
-rw------- 1 yury yury 1675 Dec 17 15:23 {192.168.100.9}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 {192.168.100.9}.pem
-rw-r--r-- 1 yury yury 1098 Dec 17 15:23 ca.csr
-rw------- 1 yury yury 1675 Dec 17 15:23 ca-key.pem
-rw-rw-r-- 1 yury yury 1505 Dec 17 15:23 ca.pem
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 peer-{192.168.100.11}.csr
-rw------- 1 yury yury 1679 Dec 17 15:23 peer-{192.168.100.11}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 peer-{192.168.100.11}.pem
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 peer-{192.168.100.12}.csr
-rw------- 1 yury yury 1675 Dec 17 15:23 peer-{192.168.100.12}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 peer-{192.168.100.12}.pem
-rw-r--r-- 1 yury yury 1066 Dec 17 15:23 peer-{192.168.100.9}.csr
-rw------- 1 yury yury 1675 Dec 17 15:23 peer-{192.168.100.9}-key.pem
-rw-rw-r-- 1 yury yury 1537 Dec 17 15:23 peer-{192.168.100.9}.pem

Copy certs files

for u2004s01 run the conmmands

yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/ca.pem [email protected]:/etc/etcd/ca.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.9}.pem [email protected]:/etc/etcd/peer-{192.168.100.9}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.9}-key.pem [email protected]:/etc/etcd/peer-{192.168.100.9}-key.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.9}.pem [email protected]:/etc/etcd/{192.168.100.9}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.9}-key.pem [email protected]:/etc/etcd/{192.168.100.9}-key.pem

yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/ca.pem [email protected]:/etc/etcd/ca.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.11}.pem [email protected]:/etc/etcd/peer-{192.168.100.11}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.11}-key.pem [email protected]:/etc/etcd/peer-{192.168.100.11}-key.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.11}.pem [email protected]:/etc/etcd/{192.168.100.11}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.11}-key.pem [email protected]:/etc/etcd/{192.168.100.11}-key.pem

yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/ca.pem [email protected]:/etc/etcd/ca.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.12}.pem [email protected]:/etc/etcd/peer-{192.168.100.12}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/peer-{192.168.100.12}-key.pem [email protected]:/etc/etcd/peer-{192.168.100.12}-key.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.12}.pem [email protected]:/etc/etcd/{192.168.100.12}.pem
yury@u2004s01:~/tlssetup/hack/tls-setup$ sudo scp -r certs/{192.168.100.12}-key.pem [email protected]:/etc/etcd/{192.168.100.12}-key.pem

Start the cluster

for each u2004s01, u2004s02, u2004s03 run the command

sudo nano /etc/systemd/system/etcd.service

insert the following content for u2004s01

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s01 \
  --initial-advertise-peer-urls https://192.168.100.11:2380 \
  --listen-peer-urls https://192.168.100.11:2380 \
  --listen-client-urls https://192.168.100.11:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.11:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --peer-client-cert-auth \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-cert-file=/etc/etcd/peer-{192.168.100.11}.pem \
  --peer-key-file=/etc/etcd/peer-{192.168.100.11}-key.pem \
  --client-cert-auth \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --cert-file=/etc/etcd/{192.168.100.11}.pem \
  --key-file=/etc/etcd/{192.168.100.11}-key.pem
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s02

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s02 \
  --initial-advertise-peer-urls https://192.168.100.9:2380 \
  --listen-peer-urls https://192.168.100.9:2380 \
  --listen-client-urls https://192.168.100.9:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.9:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --peer-client-cert-auth \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-cert-file=/etc/etcd/peer-{192.168.100.9}.pem \
  --peer-key-file=/etc/etcd/peer-{192.168.100.9}-key.pem \
  --client-cert-auth \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --cert-file=/etc/etcd/{192.168.100.9}.pem \
  --key-file=/etc/etcd/{192.168.100.9}-key.pem
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

insert the following content for u2004s03

[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name u2004s03 \
  --initial-advertise-peer-urls https://192.168.100.12:2380 \
  --listen-peer-urls https://192.168.100.12:2380 \
  --listen-client-urls https://192.168.100.12:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.100.12:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster  u2004s01=https://192.168.100.11:2380,u2004s02=https://192.168.100.9:2380,u2004s03=https://192.168.100.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd \
  --peer-client-cert-auth \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-cert-file=/etc/etcd/peer-{192.168.100.12}.pem \
  --peer-key-file=/etc/etcd/peer-{192.168.100.12}-key.pem \
  --client-cert-auth \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --cert-file=/etc/etcd/{192.168.100.12}.pem \
  --key-file=/etc/etcd/{192.168.100.12}-key.pem
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

yury@u2004s01:~$ sudo etcdctl member list --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.11}.pem --key /etc/etcd/{192.168.100.11}-key.pem
yury@u2004s02:~$ sudo etcdctl member list --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.9}.pem --key /etc/etcd/{192.168.100.9}-key.pem

yury@u2004s03:~$ sudo etcdctl member list --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.12}.pem --key /etc/etcd/{192.168.100.12}-key.pem
f592fd9ac4ffa37, started, u2004s01, https://192.168.100.11:2380, https://192.168.100.11:2379, false
2c1390a86cc037db, started, u2004s03, https://192.168.100.12:2380, https://192.168.100.12:2379, false
49d66ef58fa104e5, started, u2004s02, https://192.168.100.9:2380, https://192.168.100.9:2379, false

yury@u2004s03:~$ sudo etcdctl --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.12}.pem --key /etc/etcd/{192.168.100.12}-key.pem put foo "Hello World!"
OK

yury@u2004s01:~$ sudo etcdctl --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.11}.pem --key /etc/etcd/{192.168.100.11}-key.pem   get foo
foo
Hello World!
yury@u2004s01:~$ sudo etcdctl --insecure-skip-tls-verify --cert /etc/etcd/{192.168.100.11}.pem --key /etc/etcd/{192.168.100.11}-key.pem del foo
1

Destroy the cluster

for each u2004s01, u2004s02, u2004s03 run the commands

sudo systemctl stop etcd
sudo systemctl disable etcd
sudo rm /etc/systemd/system/etcd.service

sudo ls -l /var/lib/etcd/
total 4
drwx------ 4 root root 4096 Dec 18 11:18 member

sudo rm -r /var/lib/etcd/member
sudo rm -r /etc/etcd/

sudo systemctl daemon-reload

Prometheus

read the article Prometheus

Install

for each u2004s01 run the commands

PROMETHEUS_VERSION="2.32.1"
wget https://github.com/prometheus/prometheus/releases/download/v$PROMETHEUS_VERSION/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz -O /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz
tar -xvzf /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz --directory /tmp/ --strip-components=1

yury@u2004s01:~$ /tmp/prometheus --version
prometheus, version 2.32.1 (branch: HEAD, revision: 41f1a8125e664985dd30674e5bdf6b683eff5d32)
  build user:       root@54b6dbd48b97
  build date:       20211217-22:08:06
  go version:       go1.17.5
  platform:         linux/amd64

Settings

for each u2004s01 run the command

nano /tmp/test-etcd.yaml

insert the following content

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: test-etcd
    static_configs:
    - targets: ['192.168.100.11:2379','192.168.100.9:2379','192.168.100.12:2379']

Start

for each u2004s01 run the command

nohup /tmp/prometheus -config.file /tmp/test-etcd.yaml -web.listen-address ":9090" -storage.local.path "test-etcd.data" >> /tmp/test-etcd.log  2>&1 &

or run the command (to get ready for grafana)
- and in your browser enter http://192.168.100.11:9090

/tmp/prometheus --config.file /tmp/test-etcd.yaml --web.listen-address ":9090"

Grafana

read the article Grafana

Install

read the article To install the latest Enterprise edition
for each u2004s01 run the commands

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana-enterprise

Start the server

for each u2004s01 run the commands

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Configure the Grafana server to start at boot

for each u2004s01 run the command

sudo systemctl enable grafana-server.service

Login

in your browser enter
- login as user=admin with password=admin
  - (reset admin password)

http://192.168.100.11:3000

Create DataSource

we selected Type: Prometheus
we selected URL: http://localhost:9090
we setted the datasource name: test-etcd

Import Dashboard

read the article Exporting a dashboard
url: https://grafana.com/grafana/dashboards/3070

U1.44 Ubuntu Quick Start (QS): Etcd cluster on premises - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki

Reading

We start with

Prepare u2004m01 u2004m02 u2004m03

Download and install binaries

Creating etcd folders

Static bootstrapping an etcd cluster without TLS

Test

Destroy the cluster

Static bootstrapping an etcd cluster with TLS Automatic certificates

Destroy the cluster

SelfSigned certificates

Generating Self-signed certificates

Download "tls-setup" folder

Install "make" utility

Install "golang-cfssl" utility

Modify config/req-csr.json

Generate certs

Copy certs files

Start the cluster

Destroy the cluster

Prometheus

Install

Settings

Start

Grafana

Install

Start the server

Configure the Grafana server to start at boot

Login

Create DataSource

Import Dashboard

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️