Kubernetes Production Cluster - CloudCommandos/JohnChan GitHub Wiki

Assumptions

  • All nodes are running Ubuntu 18.04
  • Cluster Setup:
VM Name Role
etcd1 etcd
etcd2 etcd
etcd3 etcd
master1 k8s master
master2 k8s master
master3 k8s master
worker1 k8s worker
worker2 k8s worker
worker3 k8s worker
worker... k8s worker

Prerequisites

  • Use root user for all operations in this guide.
  • Disable swap for all nodes. (run swapoff -a and modify /etc/fstab)
  • First K8s master node should have SSH access to all nodes for this setup process.
  • First etcd node should have SSH access to all etcd nodes for this setup process. (SSH key access is recommended)

Setup Etcd Cluster

Run the following bash script on the first etcd node. Modify the #etcd cluster settings accordingly.

#!/bin/bash
# RUN THIS SCRIPT ON THE FIRST ETCD NODE
# Make sure that swap is off for all nodes

#colours
COLOUR1='\033[1;33m'
COLOUR2='\033[1;36m'
NC='\033[1;37m'

#etcd cluster settings
ETCD_NODE=("etcd1" "etcd2" "etcd3")
ETCD_INTERNAL_IP=("10.0.1.201" "10.0.1.202" "10.0.1.203")
USER=root  #Do not change this

# Install prerequisites
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
#  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
EOF
for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
  echo -e "${COLOUR1}Installing prerequisites on ${COLOUR2}${ETCD_NODE[$i]}${COLOUR1} ...${NC}"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} " \
    apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} " \
    add-apt-repository \
    'deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable'"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get install -y docker-ce docker-ce-cli containerd.io"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get install -y iptables arptables ebtables"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} " \
    update-alternatives --set iptables /usr/sbin/iptables-legacy; \
    update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy; \
    update-alternatives --set arptables /usr/sbin/arptables-legacy; \
    update-alternatives --set ebtables /usr/sbin/ebtables-legacy; \
    "
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"
  scp /etc/apt/sources.list.d/kubernetes.list ${USER}@${ETCD_INTERNAL_IP[$i]}:/etc/apt/sources.list.d/kubernetes.list
  scp /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf ${USER}@${ETCD_INTERNAL_IP[$i]}:/etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-get install -y kubelet kubeadm"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "apt-mark hold kubelet kubeadm"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "sed -i '/ExecStart=\/usr\/bin\/dockerd -H fd:\/\/ --containerd=\/run\/containerd\/containerd.sock/c\\
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd' /lib/systemd/system/docker.service"
done

# Create temp directories to store files that will end up on other hosts.
for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
  mkdir -p /tmp/${ETCD_INTERNAL_IP[$i]}
done


tmp_initial_cluster_str=''
for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
  if [ "$tmp_initial_cluster_str" == '' ]
  then
    tmp_initial_cluster_str="${ETCD_NODE[$i]}=https://${ETCD_INTERNAL_IP[$i]}:2380"
  else
    tmp_initial_cluster_str="${ETCD_NODE[$i]}=https://${ETCD_INTERNAL_IP[$i]}:2380,${tmp_initial_cluster_str}"
  fi
done
echo "etcd cluster: $tmp_initial_cluster_str"


for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
echo -e "${COLOUR1}Generating etcd cluster config for ${COLOUR2}${ETCD_NODE[$i]}${COLOUR1} ...${NC}"
cat << EOF > /tmp/${ETCD_INTERNAL_IP[$i]}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${ETCD_INTERNAL_IP[$i]}"
        peerCertSANs:
        - "${ETCD_INTERNAL_IP[$i]}"
        extraArgs:
            initial-cluster: ${tmp_initial_cluster_str}
            initial-cluster-state: new
            name: ${ETCD_NODE[$i]}
            listen-peer-urls: https://${ETCD_INTERNAL_IP[$i]}:2380
            listen-client-urls: https://${ETCD_INTERNAL_IP[$i]}:2379
            advertise-client-urls: https://${ETCD_INTERNAL_IP[$i]}:2379
            initial-advertise-peer-urls: https://${ETCD_INTERNAL_IP[$i]}:2380
EOF
done

echo -e "${COLOUR1}Initializing etcd CA cert ...${NC}"
kubeadm init phase certs etcd-ca

for ((i=${#ETCD_INTERNAL_IP[@]} - 1; i>=0;--i))
do
  echo -e "${COLOUR1}Generating certs for ${COLOUR2}${ETCD_NODE[$i]}${COLOUR1} ...${NC}"
  kubeadm init phase certs etcd-server --config=/tmp/${ETCD_INTERNAL_IP[$i]}/kubeadmcfg.yaml
  kubeadm init phase certs etcd-peer --config=/tmp/${ETCD_INTERNAL_IP[$i]}/kubeadmcfg.yaml
  kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${ETCD_INTERNAL_IP[$i]}/kubeadmcfg.yaml
  kubeadm init phase certs apiserver-etcd-client --config=/tmp/${ETCD_INTERNAL_IP[$i]}/kubeadmcfg.yaml
  
  cp -R /etc/kubernetes/pki /tmp/${ETCD_INTERNAL_IP[$i]}/
  # cleanup non-reusable certificates
  find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
  # clean up certs that should not be copied off this host
  if [[ "$i" != "0" ]]
  then
    find /tmp/${ETCD_INTERNAL_IP[$i]} -name ca.key -type f -delete
  fi
done

for ((i=${#ETCD_INTERNAL_IP[@]} - 1; i>=0;--i))
do
  echo -e "${COLOUR1}Transferring certs to ${COLOUR2}${ETCD_NODE[$i]}${COLOUR1} ...${NC}"
  scp -r /tmp/${ETCD_INTERNAL_IP[$i]}/* ${USER}@${ETCD_INTERNAL_IP[$i]}:
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "chown -R root:root pki; rm -rf /etc/kubernetes/pki; mv pki /etc/kubernetes/"
done

for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
  echo -e "${COLOUR1}Initializing etcd on ${COLOUR2}${ETCD_NODE[$i]}${COLOUR1} ...${NC}"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml"
  ssh ${USER}@${ETCD_INTERNAL_IP[$i]} "systemctl daemon-reload;systemctl restart docker; systemctl restart kubelet;"
done

##############################################################################################################################
#HEALTH CHECKS
echo -e "${COLOUR1}Commencing health check after 5s ...${NC}"
sleep 5s
kubelet_version=$(kubelet --version 2>/dev/null | cut -d'v' -f2)
etcd_version=$(kubeadm config images list --kubernetes-version ${kubelet_version} 2>/dev/null | grep etcd | cut -d':' -f2)
for ((i=0; i<${#ETCD_INTERNAL_IP[@]};++i))
do
  docker run --rm -it \
  --net host \
  -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:${etcd_version} etcdctl \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --endpoints https://${ETCD_INTERNAL_IP[$i]}:2379 endpoint health --cluster
done

echo -e "${COLOUR1}Done!${NC}"

Optional: Delete all etcd keys

Before using the etcd cluster for a new k8s cluster you should delete away old keys on etcd.

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl \
--cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
--key /etc/kubernetes/pki/apiserver-etcd-client.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://10.0.1.201:2379 del "" --from-key=true

Setup Load Balancer

For this guide, the Load Balancer(s) is represented by the IP 10.0.1.99. It is configured to redirect traffic coming to 10.0.1.99:6443 to the k8s master node's port 6443 (kube-apiserver).

Setup Multi-Master K8s Cluster

Run the following bash script on the first k8s master node to install Docker, Kubelet, kubeadm, and kubectl on all k8s nodes. Modify the #k8s cluster settings accordingly.

#!/bin/bash
# RUN THIS SCRIPT ON THE FIRST K8S MASTER NODE
# Make sure that swap is off for all nodes

#colours
COLOUR1='\033[1;33m'
COLOUR2='\033[1;36m'
NC='\033[1;37m'

#k8s cluster settings
K8S_NODE=("master1" "master2" "master3" "worker1" "worker2" "worker3")
K8S_INTERNAL_IP=("10.0.1.111" "10.0.1.112" "10.0.1.113" "10.0.1.114" "10.0.1.115" "10.0.1.116")
USER=root  #Do not change this

# Install prerequisites
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
for ((i=0; i<${#K8S_INTERNAL_IP[@]};++i))
do
  echo -e "${COLOUR1}Installing prerequisites on ${COLOUR2}${K8S_NODE[$i]}${COLOUR1} ...${NC}"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} " \
    apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} " \
    add-apt-repository \
    'deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable'"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get install -y docker-ce docker-ce-cli containerd.io"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get install -y iptables arptables ebtables"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} " \
    update-alternatives --set iptables /usr/sbin/iptables-legacy; \
    update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy; \
    update-alternatives --set arptables /usr/sbin/arptables-legacy; \
    update-alternatives --set ebtables /usr/sbin/ebtables-legacy; \
    "
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"
  scp /etc/apt/sources.list.d/kubernetes.list ${USER}@${K8S_INTERNAL_IP[$i]}:/etc/apt/sources.list.d/kubernetes.list
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get update"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-get install -y kubelet kubeadm kubectl"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "apt-mark hold kubelet kubeadm kubectl"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "sed -i '/ExecStart=\/usr\/bin\/dockerd -H fd:\/\/ --containerd=\/run\/containerd\/containerd.sock/c\\
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd' /lib/systemd/system/docker.service"
  ssh ${USER}@${K8S_INTERNAL_IP[$i]} "systemctl daemon-reload;systemctl restart docker;"
done

echo -e "${COLOUR1}Done!${NC}"

Copy some PKI certs from first etcd node to the first k8s master node

master1:~# mkdir -p /etc/kubernetes/pki/etcd
master1:~# scp etcd1:/etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.crt
master1:~# scp etcd1:/etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt
master1:~# scp etcd1:/etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key

Create k8s cluster initialization config, name the file 'kubeadm_config.yml'. The below config is for a k8s cluster that uses Calico as the networking implementation (192.168.0.0/16), you can change the podSubnet according to requirements of your network implementation.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "10.0.1.99:6443"
networking:
  podSubnet: 192.168.0.0/16
etcd:
    external:
        endpoints:
        - https://10.0.1.201:2379
        - https://10.0.1.202:2379
        - https://10.0.1.203:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
apiServer:
  certSANs:
  - "10.0.1.111"
  - "10.0.1.112"
  - "10.0.1.113"

Initialize k8s cluster

master1:~# kubeadm init --config kubeadm_config.yml --upload-certs

Save the join commands somewhere!

Make kubeconfig persistent

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

K8s Networking Implementation (Calico)

Download Calico k8s template

master1:~# wget https://docs.projectcalico.org/v3.12/getting-started/kubernetes/installation/hosted/calico.yaml

Run this script to configure external etcd tls connection for Calico. Change the etcd node IP accordingly.

#!/bin/bash
file="calico.yaml"
ETCD_INTERNAL_IP=("10.0.1.201" "10.0.1.202" "10.0.1.203")

etcd_endpoints_str=''
for etcd_ip in ${ETCD_INTERNAL_IP[*]}
do
  if [ "$etcd_endpoints_str" == "" ]
  then
    etcd_endpoints_str="https://${etcd_ip}:2379"
  else
    etcd_endpoints_str="https://${etcd_ip}:2379,$etcd_endpoints_str"
  fi
done
sed -i "/ etcd-ca: /c\  etcd-ca: \"$(cat /etc/kubernetes/pki/etcd/ca.crt | base64 -w 0)\"" ${file}
sed -i "/ etcd-cert: /c\  etcd-cert: \"$(cat /etc/kubernetes/pki/apiserver-etcd-client.crt | base64 -w 0)\"" ${file}
sed -i "/ etcd-key: /c\  etcd-key: \"$(cat /etc/kubernetes/pki/apiserver-etcd-client.key | base64 -w 0)\"" ${file}

sed -i "/ etcd_endpoints: /c\  etcd_endpoints: \"${etcd_endpoints_str}\"" ${file}
sed -i '/ etcd_ca: /c\  etcd_ca: "/calico-secrets/etcd-ca"' ${file}
sed -i '/ etcd_cert: /c\  etcd_cert: "/calico-secrets/etcd-cert"' ${file}
sed -i '/ etcd_key: /c\  etcd_key: "/calico-secrets/etcd-key"' ${file}

sed -i '/--cert-file=/c\          - \"--cert-file=/calico-secrets/etcd-cert\"' ${file}
sed -i '/--key-file=/c\          - \"--key-file=/calico-secrets/etcd-key\"' ${file}
sed -i '/--ca-file=/c\          - \"--ca-file=/calico-secrets/etcd-ca\"' ${file}

echo -e "Done!"

Apply Calico into K8s

master1:~# kubectl apply -f calico.yaml

Form the k8s cluster
Use the k8s join commands that you have saved previously. Control plane join commands should be used to join k8s master nodes.

⚠️ **GitHub.com Fallback** ⚠️