Kubernetes administrator

Introduction

Purpose

Describe the administration aspects of Kubernetes.

References

Kubernetes documentation
Kubernetes source code
Release note
Yaml references?
Nan20: k8s admin course.
Lig21: Kubernetes Multi Master Setup with LoadBalancer on Ubuntu
Say20 - Mastering kubernetes third edition by Gigi Sayfan, 2020
KVM k8s the hard way
k8s the hard way

Exam references

Vocabulary

ClusterRole
CNI - Container Networking Interface
Container run-time - containerd,
CRD - Custom Resource DefinitionsKubernetes Operator simply explained in 10 mins, Unlocking the power of Kubernetes: Create your own resources with CRDs
- Extend the k8s API
CRI - Container Runtime Interface.
- An interface that allows k8s to talk to the container run-time, like docker or (Nan20,2.7)
dockershim - a layer that talks to docker runtime and presents as a CRI to kubelet(no longer available as of 1.23)
EndpointSlices - the mechanism that Kubernetes uses to let your Service scale to handle large numbers of backends, and allows the cluster to update its list of healthy backends efficiently.
HA - high availability
High availability - multiple CP nodes.
- With stacked control plane nodes, where etcd nodes are colocated with control plane nodes
- With external etcd nodes, where etcd runs on separate nodes from the control plane
Ingress - lets you map (http/https?) traffic to different backends based on rules you define via the Kubernetes API.
Ingress Controllers - In order for an Ingress to work in your cluster, there must be an ingress controller running.
pki - public key infrastructure
RBAC - Role Based Access Control
Service - Expose an application running in your cluster behind a single outward-facing endpoint, even when the workload is split across multiple backends.
Role -
service mesh - (Envoy, Istio, and linkerd)
- Envoy - a modular and extensible proxy favored due to its modular constructio.
- Istio - a powerful tool set which leverages Envoy proxies via a multi-component control plane
- linkerd - purposely built to be easy to deploy, fast, and ultralight.
storage class - a designation that a can apply to a persistent volume to indicate to the user the type/speed/quality of the storage Storage Classes. k8s does not have an opinion on what classes mean.

Overview

k8s components

etcd - database with all the cluster information
- must be backed up
- handle with care during upgrade
kube-apiserver - is central to the operation of the Kubernetes cluster. All calls, both internal and external traffic, are handled via this agent
kube-scheduler - uses an algorithm to determine which node will host a Pod of containers.
kube-controller-manager - is a core control loop daemon which interacts with the kube-apiserver to determine the state of the cluster
- If the state does not match, the manager will contact the necessary controller to match the desired state.
cloud-controller-manager (ccm) - [optional] interacts with agents outside of the cloud.
CoreDNS - handles dns requests
kubelet - interacts with the underlying container engine also installed on all the nodes, and makes sure that the containers that need to run are actually running.
- Uses PodSpec
- Mounts volumes to Pod
- Downloads secrets
- Passes request to local container engine
- Reports status of Pods and node to cluster.
kube-proxy - is in charge of managing the network connectivity to the containers.
- It does so through the use of iptables entries.
- It also has the userspace mode, in which it monitors Services and Endpoints using a random port to proxy traffic via ipvs.
Supervisord - [optional] is a lightweight process monitor used in traditional Linux environments to monitor and notify about other processes.
Fluentd - [optional] data collector for unified logging layer.
service - connects resources together and will reconnect, should something die and a replacement is spawned.
- A service is an operator which listens to the endpoint operator to provide a persistent IP for Pods.
- Connect Pods together
- Expose Pods to Internet
- Decouple settings
- Define Pod access policy.
Pods
- There is only one IP address per Pod, for almost every network plugin.
- If there is more than one container in a pod, they must share the IP.
- To communicate with each other, they can either use IPC, the loopback interface, or a shared filesystem.
sidecar - name for a container dedicated to performing a helper task, like handling logs and responding to requests

k8s tools

calicoctl
etcdctl

PODs

TODO do drawing

Pods are there to abstract containers

if you had to keep track of all ports allocated for containers, you would have a mess(Nan20, 2.16)

One IP address per pod(Nan20, 2.16)
- you only have to worry about the port allocation inside the pod.
- All your mysql pod would be available on 5342, because each pad has a different IP address.
Configuration is on the pod level(Nan20, 2.16)

How the network works inside

Sandbok container TODO

Pod to Pod communications(cni)

See: AddOns

k8s does not come with a default communications solution(Nan20, 2.17)
k8s Expects you to implement the networking solution, but imposes fundamental requirements on any implementation to be pluggable into kubernets(Nan20, 2.17)
- CNI - Container Networking Interface

k8s requirements for CNI plugins(Nan20, 2.17)

every pod gets its own unique IP address
Pods on the same node can communicate with that IP address.
Pods must be able to communicate with each other accross nodes, without using NAT.

Modules:

Suggestions for K8s CNI
Benchmark results of Kubernetes network plugins (CNI) over 10Gbit/s network
calico
- 2020-08 MTU has to be manually tuned.
  - takes up abbout 6% of the cpu and provides 9500 mbits/s TCP
  - takes up abbout 10%
- Security: wireguard
  - CPU: 38%
- I need a CNI for my standard cluster
- Installing calico
cilium - has a steep learning curve
- Secuirty: IPsec
  - CPU: 11%
- I need a CNI for my (very) large-scale cluster
flannel
- I need a CNI for extra small nodes cluster OR I don’t care about security
weaveworks - latest version: 2.8.1 2021-01-25
- Secuirty: IPsec
Antrea
- Secuirty: IPsec
vmware NSX

How it works

Each node gets an IP address from the IP range of our VPC(Nan20, 2.17)
On each node a private network with a different IP range is created(Nan20, 2.17)
- The IP address range of the Nodes and the PODs should not overlap(Nan20, 2.17)
- The POD network is provided by a bridge
How are the IPs allocate accross the nodes?
- The CNI desides on a CIDR block for the cluster(Nan20, 2.17)
  - e.g. 10.32.0.0/12
- Each Node gets and equal subset of this IP range(Nan20, 2.17)
Pod-A on node 1 talk to Pod-C on node 3 using network gateway set-ups.
- Each node has a network gateway definition for all the other nodes, mapping the POD subnet to the Nodes IP address
weave has a pod on each node and all of these Pods talk directly to eachother(Nan20, 2.17)
- A pod on node-1 will ask all weave pods which of them knows the target Pod and that way find out how to route the message(Nan20, 2.17)

Secure communications

Who talks to who

DNS

Default use CoreDNS
kubelet automatically creates /etc/resolve.conf in each pod(Nan20, 3.9)
For each namespace a subdomain is created.
- <servicename>.<namespace>
  - e.g. test-nginx-svc.test-ns(Nan20, 3.9)
- <servicename>.<namespace>.svc
- FQDN <servicename>.<namespace>.svc.cluster.local
- Shorter names are supported due to the 'search' entry in the /etc/resolv.conf
  - search test-ns.svc.cluster.local svc.cluster.local cluster.local
If you have a problem accessing t/media/hck/3D_PRINTS/he service via the name and the name-space but the fully qualified name works, it means your cluster has an issue resolving the service name, e.g. look at the 'search' entry in /etc/resolv.conf

Service IP address

ClusterIP Type
- Default service type
- Exposes the service on a cluster-internal IP
- Service only reachable from with the cluster
- IP address range is defined in the Kube API Server configuration
  - /etc/kubernetes/manifests/kube-apiserver.yaml(Nan20, 3.10)
    - --service-cluster-ip-range=10.96.0.0/12
- kubeadm config print init-defaults
  - This show the default values kubeadm will use to initialize your cluster with(Nan20, 3.10).
- You can change the CIDR after installation
  - Just change it in /etc/kubernetes/manifests/kube-apiserver.yaml
  - kubelet will automatically detect the change and re-load it.

Setting up the VMs for use as k8s nodes

Installing kubeadm

AWS - creating instances

t3.small
Select your Region, on the top right corner
You need a VPC
EC2 > Instances > Launch an instance
- Name: k8s_master
- OS: Ubuntu
- Key pair(login) create new key pair
  - name: aws_admin
  - click 'create key pair'
- Network settings
  - allow ssh traffic from: My IP
click 'Launch instance'

Using the .pem key

mv ~/Downloads/MY_KEY.pem ~/.ssh/(Nan20, 2.3)
chmod 400 ~/.ssh/MY_KEY.pem
Grab the public key for the instances
ssh -i ~/.ssh/MY_KEY.pem DEFAULT_USER@PUBLIC_IP
- the user 'ubuntu' is the default for ubuntu, see the documentation(TODO Link)

KVM - creating instances

During installation select:

Ubnuntu Server minimized
diasble LVM ?
user name: ubuntu
- just like AWS
enable: Install OpenSSH server
virt-install --name master --memory 2048 --vcpus 2 --disk size=10 --cdrom $HOME/Downloads/ubuntu-22.04.2-live-server-amd64.iso --os-variant ubuntu-lts-latest --network default --virt-type=kvm
virt-install --name worker1 --memory 2048 --vcpus 2 --disk size=10 --cdrom $HOME/Downloads/ubuntu-22.04.2-live-server-amd64.iso --os-variant ubuntu-lts-latest --network default --virt-type=kvm
virt-install --name worker2 --memory 2048 --vcpus 2 --disk size=10 --cdrom $HOME/Downloads/ubuntu-22.04.2-live-server-amd64.iso --os-variant ubuntu-lts-latest --network default --virt-type=kvm

or use virt-clone

Put the virtual images on a bigger disk

sudo mkdir /hdd1/virt_image
Follow #### Create a directory based storage pool
sudo mkdir /hdd1/isos
sudo chown libvirt-qemu /hdd1
virt-install --name base_ubuntu_2204 --memory 2048 --vcpus 4 --disk size=20 --cdrom /hdd1/isos/ubuntu-22.04.2-live-server-amd64.iso --os-variant ubuntu-lts-latest --network default --virt-type=kvm
virt-clone --connect qemu:///system --original base_ubuntu_2204 --name alpha_master --file /hdd1/virt_images/alpha_master.qcow2
virt-sysprep --hostname alphamaster -a /hdd1/virt_images/alpha_master.qcow2
sudo virt-sysprep -d alpha_master --hostname alphamaster
boot
login
sudo ssh-keygen -A
sudo systemctl restart ssh
systemctl status ssh
exit
ssh cadm@IP_ADDR
sudo apt update
sudo apt install -y vim htop
sudo vi /etc/fstab
- remove the swap entry
sudo reboot
wait a few secods
ssh cadm@IP_ADDR
sudo kubeadm init
- for a single node control plane

For each VM

on the console for the VM; login as ubuntu
run ip a
from the host terminal run: ssh-copy-id ubuntu@VM_IP

Deploying k8s

Overview of deployment

First step: Generate static pod manifests for(Nan20, 2.5)
- api server/container
- sched server
- c-m (Control)
- etcd
- Put the manifests into /etc/kubernetes/manifests/
Certificates
- Generate self-signed CA certificate for k8s("Cluster root CA")(Nan20, 2.5)
- Sign all client and server certificates with it
  - Certificates are stored in /etc/kubernetes/pki/
  - Server certificate for API server endpoint
  - Client certificate for Scheduler and Controller Manager
  - Server certificate for etcd and kubelet
  - client certificate for API server to talk to kubelets and etcd
  - client certificates for k8s/media/hck/3D_PRINTS/ admins
Install CNI
Follow the steps from: kubeadm installation

prep on AWS

In the dashboard open the ports(Nan20,2.6)
- required ports
- Find the node network(of the VPC)
  - click on the instance
  - select the 'Networking' tab
  - click the VPC ID link
  - Look at the IPv4 CIDR, e.g. 172.31.0.0/16
- click the master instance
  - choose the 'Security' tab
  - click the security groups link
  - click 'edit inbound rules'
  - 6443 - external
  - 10250-10252 - internal
- security group for workers
  - 10250 - internal
  - 30000-32767 - external

Installation

ssh into each server
- sudo swapoff -a
- if you need to be able to reboot the machine:
  - sudo vi /etc/fstab
    - comment out the 'swap' line.
  - sudo reboot
  - sudo swapon --show
- sudo apt install -y vim
- update /etc/hosts with the internal IP addresses and the hostnames
  - clusterssh --options '-i ~/.ssh/AWS_KEY_PAIR.pem' --username ubuntu IP_ADDR_1 IP_ADDR_2 IP_ADDR_3
  - ip a
  - sudo vi /etc/hosts
- Change the hostname on each VM
  - sudo hostnamectl set-hostname XXX
prep the network conf, follow ip forward setup
Install the container runtime, on each server
- container-runtimes
  - sudo apt install -y containerd
  - sudo mkdir -p /etc/containerd/
  - containerd config default | sudo tee /etc/containerd/config.toml
    - config.toml
  - sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
    - update containerd
  - sudo systemctl restart containerd
set-up basic k8s tools on each node install-kubeadm
- sudo apt-get update
- sudo apt-get install -y apt-transport-https ca-certificates curl
- curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
- echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
- sudo apt-get update
- sudo apt-get install -y kubelet kubeadm kubectl
- sudo apt-mark hold kubelet kubeadm kubectl
set-up master server
- sudo kubeadm init --pod-network-cidr=[server-ip]/16
  - e.g. sudo kubeadm init --pod-network-cidr=10.42.0.0/16
- sudo kubectl get node --kubeconfig /etc/kubernetes/admin.conf
  - Status: 'NoReady' which will be fixed later
- mkdir -p ~/.kube
- sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config
- sudo chown ansible:ansible ~/.kube/config
- kubectl get node
- kubectl get namespaces
- Install CNI(Nan20, 2.18)
  - use calico
    - kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
    - vi /home/ansible/calico-custom-resources.yaml
      - for content see the 'Calico config example' section
      - in vi write:
        
        ':set paste' + enter
        
        'i' + enter
        
        at the buttom it should write: '-- INSERT (paste) --'
    - kubectl create -f /home/ansible/calico-custom-resources.yaml
    - watch kubectl get pods -n calico-system
      - it tages about 4 minutes
install and join worker nodes
- on the master run: kubeadm token create --print-join-command
  - if the kubeadm init token has run out.
- Install the worker
  - run through the same steps as for the control plane, do not execute the kubeadm init command
    - the steps must include the installation of the 'containerd'
    - starting from the beginning
- on worker1 run the kubadm join output from above, with sudo.
  - e.g sudo kubeadm join 192.168.122.161:6443 --token XXXX--discovery-token-ca-cert-hash sha256:XXXXXX
- on the control plane run watch kubectl get nodes until the worker is ready(about 2min)
- then on the control plane run: kubectl get pod -A -o wide
- also join worker2
verify weave-net
- kubectl get pod -A | grep weave-net
- kubectl logs weave-net-br5ns -n kube-system -c weave
- If you get connection problems, it is probably due to the port not having the port opened in the network security group?
check weave-net status
- kubectl get pod -n kube-system -o wide | grep weave
- kubectl exec -n kube-system weave-net-zhrgv -c weave -- /home/weave/weave --local status
  - master have 0 targets, worker1 1 targer and worker2 2 targets.
  - The important is that it has two connections and two connected.
test deployment(Nan20, 2.19)
- on master:
  - kubectl run test --image=nginx
  - kubectl get pod -w
  - kubectl get pod -o wide
  - kubectl run test2 --image=nginx
  - you should see the second test comming up on the other worker.
Copy the cluster and user informatio to your laptop
run the diagnostic tool Sonobuoy
- git clone https://github.com/vmware-tanzu/sonobuoy.git
- cd sonobuoy
- git checkout tags/v0.57.1
- sudo apt install make golang-go
- make build
  - if the build fails you might need to get the latest go version from: GO All releases
    - download the tgz, unpack it in home and append the path export PATH=${PATH}:$HOME/go/bin
    - apt-cache search go and install a newer version.
- ./sonobuoy version
- ./sonobuoy run --mode quick --wait
  - This takes about 10 minutes.
  - ./sonobuoy logs
- ./sonobuoy retrieve
  - retrieves a .tar.gz of the logs
  - ./sonobuoy results 202305191219_sonobuoy_cb9dd66e-0d22-4103-b881-04f4e6b6badf.tar.gz
- ./sonobuoy status
- ./sonobuoy delete --wait
- ./sonobuoy run --wait
Add loadbalancer and external access

Calico config example

Save it during the installation of the Controlplane

# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 10.42.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()

---

# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}

kubeadm init output

sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
root@master:/tmp# sudo kubeadm init
[init] Using Kubernetes version: v1.26.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.122.205]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.122.205 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.122.205 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.002184 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 2c1qmz.cch9oh64eaprproo
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.122.205:6443 --token 2c1qmz.cch9oh64eaprproo \
  --discovery-token-ca-cert-hash sha256:2a38d29b4d63a7d54e3095d219a7cecc6e7f8e01bd9ac951e49f9e17a26326de

kubeadm join output

ubuntu@worker1:~$ sudo kubeadm join 192.168.122.205:6443 --token ou8rw7.4w18xqujagv4va5g --discovery-token-ca-cert-hash sha256:2a38d29b4d63a7d54e3095d219a7cecc6e7f8e01bd9ac951e49f9e17a26326de
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

kubctl get node output

NAME     STATUS     ROLES           AGE   VERSION
master   NotReady   control-plane   21m   v1.26.3

Installing k8s the hard way

Kubernetes The Hard Way on KVM

Upgrade

Updating certificates

(Nan20, 18.2)

ca.crt - generated by kubeadm
Notes renewing certificates
- kubeadm renews certrificates during the cluster upgrade(Nan20, 18.2)
sudo kubeadm certs check-expiration
- (Nan20, 18.2)
- alternative command showing more details: openssl x509 -text -noout -in /etc/kubernetes/pki/ca.crt
- openssl x509 -text -noout -in /etc/kubernetes/pki/ca.crt | grep Validity -A2

Manually renewing k8s certificates

sudo kubeadm certs renew apiserver
- (Nan20, 18.3)
- What affect does this have? Does it affect all kubectl clients???

Updating k8s

Give access to the new minor version of k8s
Upgrade description
When or how often should you upgrade(Nan20, 16.2)
- Fix in later version, which affect your cluster
- Keep your cluster up-to date.
  - k8s only support the last 3 versions
    - v1.21 can upgrade v1.20 and v1.19 but NOT version v1.18
  - You should upgrade one version at a time

Process

backup the etc Operating etcd clusters for Kubernetes
upgrade the VM OS
- sudo apt update
- sudo apt upgrade
First upgrade all the Control plane node(s)(Nan20, 16.2)
- Upgrade the kubeadm tool
- run kubeadm upgrade
- kubectl drain master
  - evict all pods safely
  - mark the node as unscheduable
- upgrade kubelet
- upgrade kubectl
- kubectl uncordon master
  - change master back to schedulable
Upgrade each worker node one by one
- Upgrade the kubeadm tool
- run kubeadm upgrade
- drain the node
- upgrade kubelet
- change worker node back to schedulable
Upgrade containerd(TODO what is the generic name)
Upgrade the CNI?
What are we upgrading
- standard components
  - kube-apiserver(must be newest)
  - controller-manager(can be one version earlier)
  - kube-scheduler(can be one version earlier)
  - kubelet(can be two versions earlier)
  - kube-proxy(can be two versions earlier)
  - kubectl(can be same, one lower or one higher)
- adittional components, upgradede by kubeadm
  - etcd
  - coredns
- components installed seperately
  - weave-net
kubectl cordon XXX
- only marks the node as unschedulable
- existing pods will stay on the node

Backup etcd

See the section below: 'snapshot backup with authentication'

Upgrade the control-plane nodes

upgrading kubeadm clusters

login to the control plane node, e.g. cp0
sudo apt update
sudo apt upgrade
it seems that the version is locked to minor version of the version that is installed, so the apt repo list has to be updated see: Installing kubeadm, kubelet and kubectl
- echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
apt-cache madison kubeadm
kubeadm version
sudo apt-mark unhold kubeadm
sudo apt update
sudo apt install -y kubeadm=1.30.1-1.1`
- Why aren't we taking .1 instead of .0?
  - Oh it seems to pick that automatically
sudo apt-mark hold kubeadm
kubeadm version
sudo kubeadm upgrade plan
kubeadm upgrade apply v1.27.1
- for some reason she is forcing the .0 instead of .1
- and that failed.
- on any aditional masters you must use: sudo kubeadm upgrade node
exit
kubectl get pod -n kube-system
- shows coredns, apiserver and controlle have been updated
kubectl get node
- still shows no upgrade, because the kubelet haven't been updated yet.
kubectl drain master --ignore-daemonsets
kubectl get nodes
sudo apt-mark unhold kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubelet=1.27.1-00 kubectl=1.27.1-00
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon
kubectl get nodes

root@master:~# kubeadm upgrade apply v1.27.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.27.0"
[upgrade/versions] Cluster version: v1.26.3
[upgrade/versions] kubeadm version: v1.27.0
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W0418 11:57:01.216529  176103 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0418 11:57:40.137442  176103 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.27.0" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
W0418 11:58:11.384671  176103 staticpods.go:305] [upgrade/etcd] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0418 11:58:11.465273  176103 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-18-11-57-58/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2697268188"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-18-11-57-58/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher

[ERROR CreateJob]: Job "upgrade-health-check-qttjg" in the namespace "kube-system" did not complete in 15s: no condition of type Complete

k get pod -n kube-system | grep health | awk '{print $1}' | xargs -tn1 kubectl -n kube-system logs -f
k get pod -n kube-system | grep health | awk '{print $1}' | xargs -tn1 kubectl -n kube-system describe pod

Problem might:

either the IOP was stuffed
- I think this is the one.
- When I only ran the 3 CP and 2 workers it worked.
or the ram/cpu was overloaded

sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade/health] FATAL: [preflight] Some fatal errors occurred:
        [ERROR CreateJob]: Job "upgrade-health-check-qttjg" in the namespace "kube-system" did not complete in 15s: no condition of type Complete
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

Upgrade the worker nodes

upgrading worker nodes

login to the worker node
sudo -i
apt update
apt upgrade
apt-mark unhold kubeadm
apt-get update && apt-get install -y kubeadm=1.27.1-00
apt-mark hold kubeadm
kubeadm version
kubeadm upgrade node
On the controle-plane node run: kubectl drain <WORKER_NODE> --ignore-daemonsets
- This might fail if there are pods running without deployment or replicasets, see below
- kubectl get pods -o wide
Back on the worker node
apt-mark unhold kubelet kubectl
apt-get update && apt-get install -y kubelet=1.27.1-00 kubectl=1.27.1-00
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
exit
exit
back on master/kubectl machine
kubectl uncordon <NODE_NAME>
kubectl get nodes

error: unable to drain node "worker1" due to error:cannot delete Pods declare no controller

To fix this you need to run kubectl drain worker1 --ignore-daemonsets --force

These pods wont be restarted

ubuntu@master:~$ kubectl drain worker1 --ignore-daemonsets
node/worker1 cordoned
error: unable to drain node "worker1" due to error:cannot delete Pods declare no controller (use --force to override): default/debug-pod, continuing command...
There are pending nodes to be drained:
 worker1
cannot delete Pods declare no controller (use --force to override): default/debug-pod

Backup and restore

(Nan20, 14)

What is stored inside etcd(Nan20, 14.2)
- deployments information
- pods information
- service information
- configmaps
- secrets
NOT in the store
- Application data is not stored in etcd(Nan20, 14.2).
etcd data storage is defined in the hostPath section of /etc/kubernetes/manifests/etcd.yaml(Nan20, 14.4)
Other options to backup/restore
- Use remote storage outside the k8s cluster(Nan20, 14.4)
  - AWS, Google cloud etc.
  - On-Premises Storage
  - Instead of hostPath, use remote volume storage configuration
- Run etcd outside k8s cluster(Nan20, 14.4)
  - instead of running etcd on the master nodes, run them outside the cluster(Nan20, 14.4)

Install ectdctl

sudo apt install etcd-client
- (Nan20, 14.2)

Backup

snapshot backup with authentication

sudo ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db
--cacert /etc/kubernetes/pki/etcd/ca.crt
--cert /etc/kubernetes/pki/etcd/server.crt
--key /etc/kubernetes/pki/etcd/server.key
- the --cacert etc is to provice authentication for accessing the etcd server.(Nan20, 14.3)
- you can find the file paths in /etc/kubernetes/manifests/kube-apiserver.yaml(Nan20, 14.3)
  - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- or /etc/kubernetes/manifests/etcd.yaml
  - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
  - --cert-file=/etc/kubernetes/pki/etcd/server.crt
  - --key-file=/etc/kubernetes/pki/etcd/server.key
check the status of the file(Nan20, 14.3)
- sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /tmp/etcd-backup.db
encrypt the file(Nan20, 14.3)
copy the .db file to the safe place(Nan20, 14.3)

check snapshot status

sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /tmp/etcd-backup.db

Restore

create restore point from the backup

sudo ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db --data-dir /var/lib/etcd-backup
sudo vim /etc/kubernetes/manifests/etcd.yaml
- replace the 'path: /var/lib/etcd' with 'path: /var/lib/etcd-backup'
- do not change the 'mountPath' in 'volumenMounts' because it references the hostPath via the name
kubectl get pod -n kube-system
- this can take a minute or two.
restart componments to ensure they are not using stale data
- kube-scheduler
- kube-controller-manager
- kubelet

Granting users access

The API server handles authentication of all the requests(Nan20, 5.2)
k8s does not manage users, but simply provides ways to provide users(Nan20, 5.2)
the admin has to manage the users.(Nan20, 5.2)
Rolebinding - link/bind a role to a user or group(Nan20, 5.2)
Roles - bound to a namespace(Nan20, 5.2)
- Developers
Cluster roles - defines resoures and permissions cluster wide(Nan20, 5.2)
- k8s admins
  - managing namespaces in a cluster
  - configuring cluster-wide volumes
  - can view nodes
  - can create nodes
  - can delete nodes
ServiceAccount
- Applications - there is a k8s component that represents an application user
- Internal (to cluster)
  - You can link a ServiceAccount ClusterRole with ClusterRolbinding
  - Monitorint apps, which collect metrics from other apps within the cluster
  - Microservices needing access only within their namespaces
- External apps
  - You can link a ServiceAccount to a Role with RoleBinding.
  - Jenkins
  - terraform???
Set-up for dev team(Nan20, 5.2)
- Create the role
- add rights
- create group with dev users in it
- link the role to the group
Create users and groups(Nan20, 5.2)
- external files for authentications
  - static token file
  - certificates
  - 3rd party identity service(e.g. LDAP)
  - Admin configures an external source

For each dev team, create a role in their Namespace

kube-apiserver --token-auth-file=/users.csv [other_options](Nan20, 5.2)

Create users and groups

Creating user access via static token file

users.csv

euyioeuuoihfgdsjldhgfjksaaguye, user_1, u1001, group1
HeuhfkldjhfklgwueHJKHHkjHlkjhL, user_2, u1002, group2
djjhfkjsklusaioulkGLjhgjlghkjh, user_3, u1005, group2

Don't know which of these to use:

kubectl config get-users
kubectl get serviceaccounts -A

Creating a clusterrole

clusterrole - clusterwide
role - namespace specific See: Role and ClusterRole
kubectl create clusterrole dev-cr --verb=get,list,create,update,delete --resource=deployments.apps,pods --dry-run=client -o yaml > dev-cr.yaml
- (Nan20, 5.9)
- 'cr' stands for 'Cluster Role'
- Resources are always listed in plurals
- lookup resource api groups
  - API Overview
  - Actually I could only find this old API overview page: API Overview 1.19
  - From there I simply update the version num,ber in the ppath and found: API Overview 1.26
You can add additional resources to the yaml files
- If you want to provide all verbs to resources you can just write: verbs: ["*"]
kubectl apply -f dev-cr.yaml
kubectl get clusterroles
kubectl describe clusterrole dev-cr
kubectl create clusterrolebinding dev-crv --clusterrole=dev-cr --user=tom --dry-run=client -o yaml > dev-crb.yaml
- crb: cluster role binding
kubectl apply -f dev-crb.yaml
kubectl describe clusterrolebinding dev-cr
kubectl --kubeconfig dev-tom.conf get pods
- works
kubectl --kubeconfig dev-tom.conf get ns
- fails, because the rolebinding is not allowed to list the namespaces
kubectl auth can-i create pod --as tom
- yes
kubectl auth can-i get node --as tom
- no
kubectl auth can-i get node --as tom
- no

Enabling user access via certificates

Manually create the certificates for each user

Nan20, 5.4

/etc/kubernetes/pki
- server certificates
  - apiserver.crt
  - apiserver.key
/etc/kubernetes/pki/etcd
- etcd certs
client certificates for services talking to the API server(Nan20, 5.4)
- (client doesn't mean kubectl specifically, just anyone talking to the control plane)
- /etc/kubernetes/kubelet.conf
- Who signed them
  - kubeadm generated a CA for the k8s cluster(Nan20, 5.4)
    - kubernetes-ca or "cluster root CA"(Nan20, 5.4)
    - etcd-ca
- Why does the client trust a non-global certificate?
  - All the clients have a copy of the k8s CA(Nan20, 5.4)
The process of signing a Client Certificate(Nan20, 5.5)
- 1 create a key-pair
- 2 Generate "Certificate Signing Request(CSR)"
- 3 Send the CSR using the k8s certificate API
- 4 k8s signs the certificates for you
  - gets put in a pending queue
- 5 k8s admin approves the certificate
  - manual stop
Demo(Nan20, 5.7)
- create client key with opsenssl
  - openssl genrsa -out dev-tom.key 2048
    - this generates an rsa private key file(2048 bit)
- create CSR for the key
  - openssl req -new -key dev-tom.key -subj "/CN=tom" -out dev-tom.csr
    - csr - certificate signing request
- Approve the CSR
  - search kubernetes documentation for certificatesigningrequest : Certificate Signing Requests
  - vim dev-tom-csr.yaml
    - Copy the content from Create CertificateSigningRequest
    - name: dev-tom
    - request: cat dev-tom.csr | base64 | tr -d "\n"
    - expirationSeconds: 8640000
      - just to have some time.
  - kubectl apply -f dev-tom-csr.yaml
    - sends the signing requests
- Get the signed certificate
  - kubectl get csr
- give permissions to create, delete and update k8s resources
  - kubectl certificate approve dev-tom
  - kubectl get csr
    - the condition should now be 'Approved,Issued'
  - get csr dev-tom -o yaml | grep certificate: | awk '{ print $2}' | base64 --decode > dev-tom.crt
- validate user permissions
- test the new user
  - kubectl cluster-info
  - kubectl --server https://192.168.122.205:6443 --certificate-authority /etc/kubernetes/pki/ca.crt --client-certificate dev-tom.crt --client-key dev-tom.key get pod
    - the .kube/config content will take precedent over the command-line options.
    - mv ~/.kube/config ./admin_config
    - retry the command.
  - create a config for tom
    - cp admin_config dev-tom.conf
    - vim dev-tom.conf
      - contexts: user: dev-tom
      - contexts: name: dev-tom@kubernetes
      - current-context: dev-tom@kubernetes
      - users: name: dev-tom
      - client-certificate-data: rename to 'client-certificate:' /home/ubuntu/dev-tom.crt
        
        ref file or base64-encoded content
      - client-key-data: rename to 'client-key:' /home/ubuntu/dev-tom.key
    - kubectl --kubeconfig dev-tom.conf get pod
    - Give the three files to Tom: dev-tom.conf, dev-tom.crt, dev-tom.key(Nan20, 5.8)
      - Tom puts copies the files to ~/.kube/
      - and mv ~/.kube/dev-tom.conf to ~/.kube/config
      - Note: if you put the crt and key inside the .conf file there is only one file to copy.
        
        use base64 on the files and call the two entries client-certificate-data: and client-key-data::w

Add service account for e.g. Jenkins

Service accounts
Configure Service Accounts for Pods
ServiceAccount permissions
Managing Service Accounts
Kubernetes Service Accounts: A Practical Guide
kubectl create serviceaccount --help
kubectl create serviceaccount jenkins --dry-run=client -o yaml > jenkins-sa.yaml
- (Nan20, 5.10)
- sa: service account
kubectl apply -f jenkins-sa.yaml
kubectl describe serviceaccount jenkins
kubectl create token jenkins > jenkins.token
- This is for some fancy short term thingy
see creation below
- Manually create a long-lived API token for a ServiceAccount
- kubectl get secret/jenkins-secret
- kubectl describe secrets/jenkins-secret
  - This dumps out the token
    - the token is NOT base64
- kubectl describe secrets/jenkins-secret | grep token:
- kubectl --server=https://192.168.122.205:6443 --certificate-authority /etc/kubernetes/pki/ca.crt --token $token get pods
  - (Nan20, 5.10)
  - error: You must be logged in to the server (Unauthorized)
    - this was because the --token stated 'token' not '$token'
cp dev-tom.conf jenkins.conf
vim jenkins.conf
- change context: user: 'jenkins'
- change context: name: 'jenkins@kubernetes'
- change current-context: 'jenkins@kubernetes'
- change users: name: 'jenkins'
- delete client-certificate:
- delete client-key:
- add user: token and then add the token
  - DO NOT USE this in real life. anyone with access to the conf file could see the token
kubectl --kubeconfig jenkins.conf get pod
- this will still error on acces but it works.
Give jenkins access rights(Nan20, 5.10)
- kubectl --kubeconfig admin_config create role cicd-role --verb=create,update,list --resource=deployments.apps,services --dry-run=client -o yaml > cicd-role.yaml
  - Create a role(not a cluster role)
  - you might need to add 'namespace:' to the 'metadata:' secction in the yaml file.
- kubectl --kubeconfig admin_config apply -f cicd-role.yaml
- kubectl --kubeconfig admin_config describe role cicd-role
- kubectl --kubeconfig admin_config create rolebinding cicd-binding --role=cicd-role --serviceaccount=default:jenkins --dry-run=client -o yaml > cicd-binding.yaml
- kubectl --kubeconfig admin_config apply -f cicd-binding.yaml
- kubectl --kubeconfig admin_config describe rolebinding
- kubectl --kubeconfig admin_config auth can-i create service --as system:serviceaccount:default:jenkins -n default
  - yes

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: jenkins-secret
  annotations:
    kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token
EOF

Error from server (BadRequest): Unable to list "/v1, Resource=pods": the server rejected our request for an unknown reason (get pods)

the --server must state: 'https' (the 's' was missing).

kubectl --server=http://192.168.122.205:6443 --certificate-authority /etc/kubernetes/pki/ca.crt --token token get pods 
Error from server (BadRequest): Unable to list "/v1, Resource=pods": the server rejected our request for an unknown reason (get pods)

example of creation and usage of serviceaccount

From: Kubernetes Service Accounts: A Practical Guide

apiVersion: v1
kind: ServiceAccount
metadata:
  name: example-serviceaccount
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: example-serviceaccount
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: nginx
  serviceAccountName: example-serviceaccount

installing kubectl in an ubuntu container

apt update
apt install -y apt-transport-https ca-certificates curl pgp
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg 
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
chmod 644 /etc/apt/sources.list.d/kubernetes.list   
apt update
apt install -y kubectl

after this you can simply run: kubectl get pods and it works

Creating user access via LDAP

Creating roles

From (Nan20, 5.2)

You can use kubectl apply to pprovide roles and kubectl get to get info

check your own access rights
- kubectl auth can-i create deployments --namespace dev
admins can check permissions of other users

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cicd-role
  # if this is for a non-default namespace you would add 'namespace: my-app'
rules:
# "" indicate the core API group
- apiGroups:
  - ""
  # resources like pods, deployments, ...
  resources:
  - services
  # allowed actions on a resource: "get" "list" (read-only) or "create", "update" (read-write)
  verbs:
  - create
  - update
  - list
  # you can define access to certain pods/serviece by specifying the names: 'resourceNames: ["myapp"]
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - update
  - list

RoleBinding

(Nan20, 5.2)

Layers of secutiry

(Nan20, 5.2)
Authorization Modes
/etc/kubernetes/manifests/kube-apiserver.yaml
- --authorization-mode
Authiticate users
Autherization check
k8s supports 4 Authorization modes
- Node
- ABAC - Attribute Based Access Control.
- RBAC - Role Based Access Control.
- Webhook

Scratchpad

Static pods
- started, and managed, by the kubelet
- kubelet monitors: /etc/kubernetes/manifests/
- cannot be controlled by the controller manager
Break Your System Constructively

install stuff

ssh to all nodes at once

clusterssh --options '-i ~/.ssh/AWS_KEY_PAIR.pem' ubuntu@IP_ADDR_1 ubuntu@IP_ADDR_2 ubuntu@IP_ADDR_3

Certs and stuff

See of in SecurityQuickNotes.md

REST API

use kubectl proxy

(Nan20, 15.2)

kubectl proxy --port=8080&
curl http://localhost:8080/api/
curl http://localhost:8080/api/v1/namespaces/default/services

REST API access withoug kubectl proxy

create a service account(to reduce the power of the account)
- kubectl create serviceaccount myscript
role and rolebinding
- kubectl apply -f script-role.yaml
- kubectl create rolebinding script-role-binding --role=script-role --serviceaccount=default:myscript
- kubectl get serviceaccount myscript -o yaml
- run kubctl create secret command below...
- kubectl get secret myscript-secret -o yaml
- decode the token:
  - echo <TOKEN> | base64 --decode | tr -d "\n"
- TOKEN=<TOKEN>
- kubectl auth can-i list deployments --as system:serviceaccount:default:myscript
conncect to the REST API
- kubectl config view | grep server:
- API_SERVER=https://192.168.122.205:6443
- curl -X get $API_SERVER/apis --header "Authorization: Bearer $TOKEN" --cacert /etc/kubernetes/pki/ca.crt
  - curl -X get $API_SERVER/api --header "Authorization: Bearer $TOKEN" --cacert /etc/kubernetes/pki/ca.crt
  - this fails (due to missing 's' in /apis) with: "message": "the server does not allow this method on the requested resource",
  - This doesn't work either: curl -X get ${API_SERVER}/api/ --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/strategic-merge-patch+json" --cacert /etc/kubernetes/pki/ca.crt
    - K8s: gets HTTP 415 for PATCH request to Kubernetes REST API server
  - for paths see e.g.: list deployments, via API
- curl -X GET $API_SERVER/apis/apps/v1/namespaces/default/deployments --header "Authorization: Bearer $TOKEN" --cacert /etc/kubernetes/pki/ca.crt

script-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: script-role
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "delete"] 
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "delete"]

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: myscript-secret
  annotations:
    kubernetes.io/service-account.name: myscript
type: kubernetes.io/service-account-token
EOF

Exams tips

Certified Kubernetes Administrator (CKA) - V1.27
- 2 hours
- 66% to pass What score is needed to pass the exam?
- Troubleshooting (30%)
  - Evaluate cluster and node logging
  - Troubleshoot cluster component failure
  - Troubleshoot networking
- Cluster Architecture, Installation & Configuration (25%)
- Services & Networking (20%)
- Workloads & Scheduling (15%)
- Storage (10%)
Curriculum
CKA exam practice

(Nan20, 20.2)

During the exam use imperative kubectl commands, instead of creating yaml file(Nan20, 20.2).

Prep(Nan20, 20.2)
- alias k=kubectl
- export do="--dry-run=client -o yaml"
- alias kdp='kubectl delete pod --force --grace-period=0'
  - k delete pod
- alias kgc='kubectl config current-context'
  - k get current context
- stup autocomplete
Every time, copy the context command and paste it into the terminal, then verify it
Check the percentage that answering the question, gives you.
You need time in the end for installation and troubleshooting
Split the question into sub tasks and execute them
for each question, verify your answer
always exit the ssh
flag question you couldn't answer
Track your score
If you have time at the end, review the answers
Don't panic
kubectl run mypod --image=nginx
kubectl create deployment nginx-deployment --image=nginx --replicas=2
kubectl expose deployment nginx-deployment --type=NodePort --name=nginx-service
- TODO look up this 'expose'
kubectl create service clusterip myservice --tcp=80:80 --dry-run=client -o yaml > myservice.yaml
kubectl create service clusterip myservice --tcp=80:80 $do > myservice.yaml
get used to using --help(Nan20, 20.2)
- kubectl create --help
generate boilerplate manifests using 'kubectl create ..... --dry-run=client -o yaml'(Nan20, 20.2)
use shortcuts for frequently used commands and options(Nan20, 20.2)
kubectl edit will store if the changes fails.
- TODO how dump yaml from active/existing Deployment
prep for exam(Nan20, 20.2)
- scale
- filter
  - display all nodes, which don't have taints "NoSchedule"
  - Display all "ready" Nodes
  - Display all pods that have Resource Requests set
  - List all pods running on worker1
- kubectl top
  - TODO look-up "error: Metrics API not available"
- kubectl config set-context
Exam resources
- CKA-practice-exercises
- CKA cert simulator
- Certified Kubernetes Administrator (CKA)
- Unlock the Secrets to Acing the CKA Exam on Your First Attempt in 2023
- Curriculum
during the exams
- Pay attention to what server you are logged in at any given time(Nan20, 20.3)
- Answers
  - k8s resources
  - text files
    - Answer are saved into you initial environment(Nan20, 20.2) SO REMEMBER TO PAY ATTTENTION to where you are at any given time
- When you answer a question then you will be given a context switch command to use. Make sure that context switch is successful before proceeding
- You can only use the official browser page, and only a single page is allowed, no other sites, and only one tab(Nan20, 20.3)
- learn how to use the official documentation(Nan20, 20.3)
  - use that as the only source(Nan20, 20.3)
- other
  - manifest files of control plane components
  - certificate creation
Addition topics?
- elk/efk
- prometheous
- security
- multi control-plane
- external connection so the cluster is accessible from the outside world

Scheduler

Scheduling Plugins - The following plugins, enabled by default, implement one or more of these extension points.(LFS258, 12) Scheduler Configuration
- ImageLocality - Favors nodes that already have the container images that the Pod runs. Extension points: score.
- TaintToleration - Implements taints and tolerations. Implements extension points: filter, preScore, score.
- NodeName - Checks if a Pod spec node name matches the current node. Extension points: filter.
- NodePorts - Checks if a node has free ports for the requested Pod ports. Extension points: preFilter, filter.
- NodeAffinity - Implements node selectors and node affinity. Extension points: filter, score.
- PodTopologySpread - Implements Pod topology spread. Extension points: preFilter, filter, preScore, score.
- NodeUnschedulable - Filters out nodes that have .spec.unschedulable set to true. Extension points: filter.
- NodeResourcesFit - Checks if the node has all the resources that the Pod is requesting.
- NodeResourcesBalancedAllocation - Favors nodes that would obtain a more balanced resource usage if the Pod is scheduled there. Extension points: score.
- queueSort
- preFilter
- filter
- postFilter
- preScore
- score
- reserve
- permit
- preBind
- bind
- postBind
- multiPoint

Custom Resource Definition(CRD)

Unlocking the power of Kubernetes: Create your own resources with CRDs
Operator pattern
Operator framework
Requires a custom controller to handle the custom resources.
Operator = CRD + Custom controller + application knowledge
writing CRDs and custom controllers by hand is tedios
Operator SDKs are there to make it easier
- scfolding
- code generation
- e.g.
  - operator-sdk init --domain oit.com --repo github.com/example/iot-operator
  - operator-sdk create api --group mygroup --version v1alpha1 --kind Sensor --resource --controller
    - --resource : generate the source code for the custom api
    - --controller : generates the controller code.

Volumes

Persistent Volumes

(Nan20, 8.2)

Persistent volume(PV), a cluster resource(Nan20, 8.2)
- Are resources(Nan20, 8.2)
  - Must exist before the pod, that uses it, is created.
- Created via YAML file
- Needs actual physical storage, like
  - local disk in the k8s cluster(Nan20, 8.2)
    - violates 2. and 3. requirements for DB persistense(see below)(Nan20, 8.2)
  - external nfs server(Nan20, 8.2)
  - cloud-storage(Nan20, 8.2)
You need to decide what type of storage you need(Nan20, 8.2)
- You need to create and manage them yourself(Nan20, 8.2)
  - back-up
  - avoid corruption
  - etc.
Persistent volumes are not namespaced(Nan20, 8.2)
- Accessible to the whole cluster(Nan20, 8.2)
- The PV is ouside of the namespaces(Nan20, 8.2)
DB storage requirements
- 1. Storage that doesn't depend on the pod lifecycle(Nan20, 8.2)
- 1. Storage must be available on all nodes(Nan20, 8.2)
- 1. Storage needs to survive even if the cluster crashes(Nan20, 8.2)
The k8s admin configures the actual storage(Nan20, 8.2)
- Based off of what the developers needs(Nan20, 8.2)
- creates the nfs-storage or cloud storage
- Creates the PV components from these storage backends(Nan20, 8.2)
Why so many abstraction
- users don't care about where the data exists, as long as it has enough space and is healthy(Nan20, 8.2)
- The user simply use PVC(Nan20, 8.2)

TODO what happens to a PV/SC when a node/pod crashes or is deleted?

Storage Class(SC)

SC provisions PV dynamically, when PVC claims it(Nan20, 8.2)
Created using YAML(Nan20, 8.2)
via "provisioner" attribute(Nan20, 8.2)
- each storage backend has its own provisioner(Nan20, 8.2)
- internal provisioner - "kubernetes.io"
  - e.g. kubernetes.io/aws-ebs
- external provisioner
- configure parameters for the storage we want to request for PV(Nan20, 8.2)
Another abstraction level
- abstracts the underlying storage provider(Nan20, 8.2)
- parameters for that storage(Nan20, 8.2)
Requested by PVC(Nan20, 8.2)

Volume examples

Example Create a PersistentVolume

PersistentVolume

Configuring PV(Nan20, 8.3)
- 1 Type of persistent volume
- 2 Capacity
- 3 Access Mode

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-pv
  labels:
    type: local
spec:
  hostPath:
    path: "/mnt/data"
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce

kubectl apply -f myapp-pv.yaml
kubectl get pv

Example Create a PersistentVolumeClaim

PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  resources:
    requests:
      storage: 500M
  accessModes:
    - ReadWriteMany

(Nan20, 8.3)

k apply -f myapp-pvc.yaml
k get pvc
TODO why is the Status Pending?

Example of using pvc in pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-db
  labels:
    app: my-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-db
  template:
    metadata:
      labels:
        app: my-db
    spec:
      containers:
      - name: mysql
        image: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: mypwd
        # This is in the container
        volumeMounts:
        - name: db-data
          mountPath: "/var/lib/mysql"
      # This is at the POD level
      volumes:
      - name: db-data
        persistentVolumeClaim:
          claimName: mysql-pvc

k apply -f mysql-deployment.yaml
k describe pod my-db-5d65d648d8-g4glj
k get pods
k describe pod my-db-5d65d648d8-g4glj
- verify the volume
look at which node the pod is running at
ssh worker(1?)
ls /mnt/data

emptyDir Volume

suitable for ulti-container pods(Nan20, 8.4)
All the containers in the pod can read and write the same files in the emptyDir volume(Nan20, 8.4)
emptyDir volume n is initially empty(Nan20, 8.4)
is first created when a pod is assigned to a node and exists as long as that pod is running on that node(Nan20, 8.4)
When the pod is removed from a node, the data is deleted permanently(Nan20, 8.4)
no PV or PVC is needed(Nan20, 8.4)

The side-car is able to see the nginx /var/log in /var/sidecar

...
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: log-data
          mountPath: /var/log
      - name: log-sidecar
        image: busybox
        volumeMounts:
        - name: log-data
          mountPath: /var/sidecar
        command: ['sh', '-c']
        args:
...
      volumes:
      - name: log-data
        emptyDir: {}

ConfigMap and Secret

local volumes(Nan20, 8.2)
not created vi PV and PVC(Nan20, 8.2)
managed by k8s(Nan20, 8.2)
used for
- configuration file for your pod(Nan20, 8.2)
- certificate file for your pod(Nan20, 8.2)

How it works

1 create ConfigMap and/or Secret component(Nan20, 8.2)
2 mount that into your pod or container(Nan20, 8.2)
- Same way as PVC(Nan20, 8.2)

Resources

kubectl get pod -o jsonpath="{range .items[]} {.metadata.name}{.spec.containers[].resources}{'\n'}"
- (Nan20, 10.3)
When a node is running out of resources and has to evict a pod, it will start with pods without resource definitions.

Metrics

Metrics For Kubernetes System Components
Resource metrics pipeline
kubectl top
Metrics server
kubectl top requires 'Metrics Server'
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
- kubernetes-sigs/metrics-server
k top nodes
- error: Metrics API not available

How to Install Kubernetes (K8s) Metrics Server Step by Step

Install on Single control plane
- wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
- vi components.yaml'
  - Add to args section of the metrics-server: - --kubelet-insecure-tls
  - add to spec section: hostNetwork: true
    - Same level as 'containers:'

Security

Bastion
LSM - Linux Security Modules

kube-bench verify CIS security settings

Installing kube-bench

wget https://github.com/aquasecurity/kube-bench/releases/download/v0.6.17/kube-bench_0.6.17_linux_amd64.deb
wget https://github.com/aquasecurity/kube-bench/releases/download/v0.6.17/kube-bench_0.6.17_checksums.txt
sha256sum -c kube-bench_0.6.17_checksums.txt
sudo apt install ./kube-bench_0.6.17_linux_amd64.deb
kube-bench version

Installing Open Policy Agent(OPA)

lsf260, 4.5

mkdir source
git clone https://github.com/open-policy-agent/gatekeeper
find . -name gatekeeper.yaml
kubectl create -f ./gatekeeper/deploy/gatekeeper.yaml
kubectl get ns
kubectl -n gatekeeper-system get pod

Networking

Network plugins
The kubernetes network model
Container Network Interface (CNI) Specification
Cluster networking
CNI - the Container Network Interface
CIDR Update in Kubernetes
The CRI is responsible for starting the CNI.
- As of 1.24 this is no longer in scope for the kubelet.
- it seems that e.g. the containerd is responsible for loading the CNI

The four areas of networking:

container-to-container(inside pods): localhost communication
pod-to-pod - Cluster networking provides communication between different Pods
pod-to-service
external-to-service
Every pod gets its own unique cluster-wide IP address.
- So each pod has full control over what ports to use.

k8s fundamental requirements to the network plugin:

pods can communicate with all other pods on all other nodes without the use of NAT.
agents on a node can communicate with all pods on that node.
- e.g. kubelet
Kubernetes IP addresses exist at the Pod scope; this makes the IP network work the same for Pods and VMs.

The Pod CIDR is different from the node cidr(Nan20, 2.17)

Pod to service

Each node runs a kube-proxy process which programs iptables rules to trap access to service IPs and redirect them to the correct backends. Networkings design

External to internal

The way this is generally implemented is to set up external load balancers (e.g. GCE's ForwardingRules or AWS's ELB) which target all nodes in a cluster. When traffic arrives at a node it is recognized as being part of a particular Service and routed to an appropriate backend Pod. This does mean that some traffic will get double-bounced on the network. Once cloud providers have better offerings we can take advantage of those.

Container Network Interface(CNI)

CNI - the Container Network Interface

CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted.

Network policies

(Nan20, 19.2)

Network Policy is defined in a NetworkPolicy resource in a k8s manifest(Nan20, 19.2)
- It configures the CNI application(Nan20, 19.2)
- The network plugin implements the network policies(Nan20, 19.2)
  - Not all network plugins support network policies, e.g. Flannel does not support network policies(Nan20, 19.2)
Configuring Network Policies(Nan20, 19.2)
- Which applications are we affecting using 'podSelector'(Nan20, 19.2)
  - if podSelector: "" is used then the network policy is applied to all pods in the given namespace(Nan20, 19.2)
- What rule types
  - ingress - incomming traffic
    - from where should the incomming traffic be allowed(Nan20, 19.2)
  - egress - outgoing traffic
    - to where should the outgoing traffic be allowed(Nan20, 19.2)
    - only define rules for outgoing traffic that the pod initiates(Nan20, 19.3)
- if the source/destination pod is in a different namespace you need to add 'namespaceSelector:' to the configuration(Nan20, 19.2).
  - be carefull that is is placed under 'podSelector' not next to 'podSelector'(Nan20, 19.2)
kubectl config set-context --current --namespace myapp
kubectl get networkpolicy

Calico CNI

Installing calico cni

sudo kubeadm init --pod-network-cidr=[server-ip]/16
- e.g. sudo kubeadm init --pod-network-cidr=10.42.0.0/16
ususal kube stuff
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml
vim custom-resources.yaml
- update the line 'cidr: e.g. cidr: 10.42.0.0/16
kubectl create -f custom-resources.yaml
watch kubectl get pods -n calico-system

Ingress

Getting Started with Kubernetes Ingress-Nginx on Minikube
Set up Ingress on Minikube with the NGINX Ingress Controller
Ingress Rewrite
Ingress also requires an ingress controller(Nan20, 4.4)
Usually the Cloud provider have the LoadBalancer that you need(Nan20, 4.4)
ingress-nginx Installation Guide
F5 NGINX Ingress Controller
kubectl describe ingress my-app-ingress

Ingress example

# https://kubernetes.io/docs/concepts/services-networking/ingress/
# https://kubernetes.github.io/ingress-nginx/examples/rewrite/
apiVersion: networking.k8s.io/v1
# Ingress type.
kind: Ingress
metadata:
  # Arbitrary name given to this ingress
  name: rest-ingress
  # Namespace in which, this rule is active
  namespace: default
  annotations:
    # the '$2' refers to the second '()' in the 'path:' entry, below
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  # This is the default classname for nginx, unless you have multiple controllers
  ingressClassName: nginx
  rules:
  # The http is the protocol used to access the server(Nan20, 4.4)
  - http:
      paths:
      # the path the external client will use
      - path: /rest(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            # name of the service this message will be forwarded to
            name: rest-service
            port:
              # Port number for the service
              number: 8000

Ingress controller

It seems that the ingress controller is a pod.

You can have multiple Ingress controllers, but you have to give each controller a unique controller-class and ingress-class, please see: Multiple Ingress controllers

erDiagram
  OutsideWorld }o--|| IngressController : http_s
  IngressController ||--|| Service_1 : path
  IngressController ||--|| Service_2 : path
  IngressController ||--|| Service_n : path
  Service_1 ||--|{ pod_alpha_1 : path
  Service_1 ||--|{ pod_alpha_2 : path
  Service_2 ||--|{ pod_bravo_1 : path
  Service_n ||--|{ pod_omega_1 : path
  Service_n ||--|{ pod_omega_2 : path
  Service_n ||--|{ pod_omega_n : path

installing the ingress-nginx

Ingress controller logs

kubectl get pods --no-headers=true -o custom-columns=NAME:.metadata.name -A | grep ingress-nginx-controller | xargs kubectl logs -n ingress-nginx
- please note the '-n ingress-nginx' might be '-n kube-system' on your system.

Ingress controller configuration

kubectl get pods --no-headers=true -o custom-columns=NAME:.metadata.name -A | grep ingress-nginx-controller | xargs -I{} kubectl exec -it -n ingress-nginx {} cat /etc/nginx/nginx.conf

Additional tools and services

Installing a docker repository in k8s

create certificate for the registry server

ssh to the cp
mkdir docker_repo
cd docker_repo
create the registry certificate
- openssl req -newkey rsa:2048 -keyout docker_registry.key -out docker_registry_request.csr -config docker_registry.cnf -nodes
- openssl req -noout -text -in docker_registry_request.csr
sign the registry certificate
- sudo openssl x509 -req -extensions v3_req -extfile sign_docker_registry.cnf -in docker_registry_request.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out docker_registry.crt -days 1000
- openssl x509 -text -in docker_registry.crt

docker_registry.cnf:

[req]
default_bits  = 2048
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName = ZZ
stateOrProvinceName = N/A
localityName = N/A
organizationName = Docker registry Self-signed certificate
commonName = request

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
IP.1 = 192.168.1.102

deploy docker repo to k8s

Private Docker Registry on Kubernetes: Steps to Set Up
kubectl create secret tls certs-secret --cert=/registry/certs/tls.crt --key=/registry/certs/tls.key

configure haproxy

rontend lfs_repo_front
   mode tcp
   bind *:5000
   log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq dstName:%[var(sess.dstName)] dstIP:%[var(sess.dstIP)] "
   default_backend lfs_repo_back
   timeout client 1h

backend lfs_repo_back
   mode tcp
   server lfs_cp 192.168.122.79:32005 check

verify with curl

cp the /etc/kubernetes/pki/ca.crt from the control plane to k8s_ca.crt curl --cacert k8s_ca.crt https://192.168.1.102:5000/v2/_catalog

set-up client for pushing to the repo

sudo mkdir -p /etc/docker/certs.d/HOST_ADDRESS:5000/
cp the /etc/kubernetes/pki/ca.crt from the control plane to /etc/docker/certs.d/HOST_ADDRESS:5000/ca.crt
docker tag alpine:3.19 192.168.1.102:5000/alpine:purple
docker push 192.168.1.102:5000/alpine:purple

fix:

sudo mv /etc/docker/certs.d/192.168.1.102 /etc/docker/certs.d/192.168.1.102:5000

The push refers to repository [192.168.1.102:5000/alpine]
Get "https://192.168.1.102:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

set-up the ca.crt

How to fix it Adding a trusted certificate for containerd on Kubernetes using a DaemonSet

cert stuff
- sudo -i
- cp /etc/kubernetes/pki/ca.crt /usr/local/share/ca-certificates/ca.crt
- update-ca-certificates
- systemctl restart containerd
openssl s_client -showcerts -connect 192.168.1.102:5000

deploy in the cluster

sudo mkdir -p /etc/docker/certs.d/192.168.1.102:5000
sudo cp /etc/kubernetes/pki/ca.crt /etc/docker/certs.d/192.168.1.102:5000/ca.crt
kubectl run alpine-purple --image=192.168.1.102:5000/alpine:purple --restart=Never -- sleep infinity

This works fine

curl --cacert /etc/docker/certs.d/192.168.1.102:5000/ca.crt https://192.168.1.102:5000/v2/alpine/manifests/purple

k describe pod alpine-purple
  Normal   Scheduled  13s   default-scheduler  Successfully assigned default/alpine-purple to worker0
  Normal   Pulling    12s   kubelet            Pulling image "192.168.1.102:5000/alpine:purple"
  Warning  Failed     12s   kubelet            Failed to pull image "192.168.1.102:5000/alpine:purple": failed to pull and unpack image "192.168.1.102:5000/alpine:purple": failed to resolve reference "192.168.1.102:5000/alpine:purple": failed to do request: Head "https://192.168.1.102:5000/v2/alpine/manifests/purple": tls: failed to verify certificate: x509: certificate signed by unknown authority
  Warning  Failed     12s   kubelet            Error: ErrImagePull
  Normal   BackOff    11s   kubelet            Back-off pulling image "192.168.1.102:5000/alpine:purple"
  Warning  Failed     11s   kubelet            Error: ImagePullBackOff

High Availability(HA)

only cp0 needs to run the kubeadm init, the other CPs run the cp join command. all other nodes, both workers adn CPs, runs the standard node installation(kubelet etc) The calico is automatically installed on all nodes that join, both for CP and worker. The certificates are automatically distributed.

on deployment machine
- terraform apply -var project_tag="ha" -var load_balancer_dns="homelab2"
  - tf apply -parallelism=1
on cp0
- sudo vi /etc/hosts
  - 192.168.1.103 homelab2
- ansible-playbook --extra-vars "kubernetes_version=v1.29" --extra-vars "cni_selection=calico" --extra-vars "node_type=control_plane" --extra-vars "load_balancer_dns=homelab2" --extra-vars "load_balancer_port=6443" test_playbook.yaml
- cat kubeadm_init.log
on cp1+
- byobu
- sudo vi /etc/hosts
  - 192.168.1.103 homelab2
- sudo kubeadm join homelab3:6443 --token j0asld.lpmtvrkll3pkor93 --discovery-token-ca-cert-hash sha256:4eff9fe177ecfb03613b7992f8f3ba5baa53c2d7989fd6930f12894e6b48a493 --control-plane --certificate-key 85fce70a5201a7213f253da53f5fb9d181e0205c1cc6868990d334891e234209
on worker*
- byobu
- sudo vi /etc/hosts
  - 192.168.1.103 homelab2
- sudo kubeadm join homelab3:6443 --token j0asld.lpmtvrkll3pkor93 --discovery-token-ca-cert-hash sha256:4eff9fe177ecfb03613b7992f8f3ba5baa53c2d7989fd6930f12894e6b48a493
on "host"
- libvirt - using haproxy

how to identify the leader:

kubectl get endpoints
- if the etcd is colocated with CP then you can just replace the port number with 2379 and you are good to go
kubectl -n kube-system get pods | grep etcd
kubectl -n kube-system exec -it etcd-cp0 -- /bin/sh

ETCDCTL_API=3 etcdctl -w table \
--endpoints 10.128.0.66:2379,10.128.0.24:2379,10.128.0.30:2379 \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/server.crt \
--key /etc/kubernetes/pki/etcd/server.key \
endpoint status

ETCDCTL_API=3 etcdctl -w table \                                                      
> --endpoints 192.168.122.106:2379,192.168.122.147:2379,192.168.122.190:2379 \
> --cacert /etc/kubernetes/pki/etcd/ca.crt \
> --cert /etc/kubernetes/pki/etcd/server.crt \
> --key /etc/kubernetes/pki/etcd/server.key \
> endpoint status
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.122.106:2379 | 181a42ad80aefbc6 |  3.5.12 |  6.4 MB |      true |      false |         3 |      42980 |              42979 |        |
| 192.168.122.147:2379 | bf11777c4e5cd90f |  3.5.12 |  6.4 MB |     false |      false |         3 |      42980 |              42980 |        |
| 192.168.122.190:2379 | 2bfaa228186c7398 |  3.5.12 |  6.5 MB |     false |      false |         3 |      42980 |              42980 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

after having run shutdown on the leader:

ETCDCTL_API=3 etcdctl -w table \
> --endpoints 192.168.122.106:2379,192.168.122.147:2379,192.168.122.190:2379 \
> --cacert /etc/kubernetes/pki/etcd/ca.crt \
> --cert /etc/kubernetes/pki/etcd/server.crt \
> --key /etc/kubernetes/pki/etcd/server.key \
> endpoint status
{"level":"warn","ts":"2024-04-24T21:03:19.681426Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00030ee00/192.168.122.106:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint 192.168.122.106:2379 (context deadline exceeded)
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.122.147:2379 | bf11777c4e5cd90f |  3.5.12 |  6.4 MB |     false |      false |         4 |      45253 |              45253 |        |
| 192.168.122.190:2379 | 2bfaa228186c7398 |  3.5.12 |  6.5 MB |      true |      false |         4 |      45254 |              45254 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

TODO find out how to find the etcd endoints this failed because I was using the 6443 port I need to use the 2379 port for etcd

ETCDCTL_API=3 etcdctl -w table \
> --endpoints 192.168.122.106:6443,192.168.122.147:6443,192.168.122.190:6443 \
> --cacert /etc/kubernetes/pki/etcd/ca.crt \
> --cert /etc/kubernetes/pki/etcd/server.crt \
> --key /etc/kubernetes/pki/etcd/server.key \
> endpoint status
{"level":"warn","ts":"2024-04-24T20:34:41.872586Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00028d180/192.168.122.106:6443","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\""}
Failed to get the status of endpoint 192.168.122.106:6443 (context deadline exceeded)
{"level":"warn","ts":"2024-04-24T20:34:46.873696Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00028d180/192.168.122.106:6443","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\""}
Failed to get the status of endpoint 192.168.122.147:6443 (context deadline exceeded)
{"level":"warn","ts":"2024-04-24T20:34:51.874725Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00028d180/192.168.122.106:6443","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\""}
Failed to get the status of endpoint 192.168.122.190:6443 (context deadline exceeded)
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+----------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Load balancer

using haproxy as a load balancer

# homelab2
# from: https://www.youtube.com/watch?v=c1SCdv2hYDc
frontend homelab2-k8s-cp-frontend
   bind homelab2:6443
   mode tcp
   option tcplog
   default_backend homelab2-k8s-cp-backend

backend homelab2-k8s-cp-backend
   mode tcp
   option tcp-check
   balance roundrobin
   server hak8scp1 192.168.122.190:6443 check fall 3 rise 2
   server hak8scp2 192.168.122.106:6443 check fall 3 rise 2
   server hak8scp3 192.168.122.147:6443 check fall 3 rise 2

Troubleshooting

Troubleshoot access issues

Please enter Username

the context references the wrong 'user'

- context:
    cluster: homelab2_k8s
    namespace: default
    user: kubernetes-admin

$ kubectl cluster-info
Please enter Username:

Troubleshooting kubelet and kubectl issues

Troubleshoot kubelet

systemctl status kubelet
- (Nan20, 6.5)
journalctl -u kubelet
which kubelet
sudo cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Troubleshoot kubectl

(Nan20 6.5)

cat ~/.kube/config
- Validate all conf are ok
Check the cluster configuration
- check the CA
  - take the content of certificate-authority-data:
  - base64 --decode it
  - compare it to the cluster /etc/kubernetes/pki/ca.crt
- kubectl config view
  - verify the ip address

troubleshooting seting up master

Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

In Nan20, 2.15 it is mentioned that this is because there is no pod networl

master status NotReady
both coredns' are pending.

it seems to be the swap that has to be permanently disabled. Nah it still fails.

Maybe this? troubleshooting-cni-plugin-related-errors

sudo swapon --show
systemctl status kubelet
sudo ls -l /var/run/containerd/containerd.sock
sudo crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause
- sudo crictl ps |grep apiserver
  - [lfs260, ch5]
- sudo cat /var/log/containers/kube-apiserver-cp-<YOUR-CONTAINER-NAME>
- tail -f /var/log/audit.log

container-runtime-network-not-ready-cni-config-uninitialized

sudo systemctl stop apparmor
sudo systemctl disable apparmor
sudo systemctl restart containerd.service

Unfortunately, an error has occurred:
  timed out waiting for the condition

This error is likely caused by:
  - The kubelet is not running
  - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
  - 'systemctl status kubelet'
  - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
  - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
  Once you have found the failing container, you can inspect its logs with:
  - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

Troubleshoot CNI

Could not resolve CalicoNetwork IPPool and kubeadm configuration

It seemed that I needed the '--pod-network-cidr' in sudo kubeadm init --pod-network-cidr=[server-ip]/16 and updating the cidr in the custom file. See calico installation above.

{"level":"error","ts":"2023-08-29T10:49:32Z","logger":"controller_installation","msg":"Error querying installation","Request.Namespace":"","Request.Name":"default","reason":"ResourceReadError","error":"Could not resolve CalicoNetwork IPPool and kubeadm configuration: kubeadm configuration is missing required podSubnet field","stacktrace":"github.com/tigera/operator/pkg/controller/status.(*statusManager).SetDegraded\n\t/go/src/github.com/tigera/operator/pkg/controller/status/status.go:406\ngithub.com/tigera/operator/pkg/controller/installation.(*ReconcileInstallation).Reconcile\n\t/go/src/github.com/tigera/operator/pkg/controller/installation/core_controller.go:872\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-29T10:49:32Z","msg":"Reconciler error","controller":"tigera-installation-controller","object":{"name":"default"},"namespace":"","name":"default","reconcileID":"6061a783-7373-4910-874c-f43856bc6789","error":"Could not resolve CalicoNetwork IPPool and kubeadm configuration: kubeadm configuration is missing required podSubnet field","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}

Troubleshoot volumes

pvc fail with no persistent volumes available for this claim and no storage class is set

I had to add storageClassName: slow to the spec: section of the volume claim.

  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: slow
      resources:
        requests:
          storage: 1Gi

Administrators can specify a default StorageClass only for PVCs that don't request any particular class to bind to: see the PersistentVolumeClaim section for details.

k get pvc
NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Pending

k describe pvc www-web-0
Name:          www-web-0
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        app=nginx
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:  
VolumeMode:    Filesystem
Used By:       web-0
Events:
  Type    Reason         Age                  From                         Message
  ----    ------         ----                 ----                         -------
  Normal  FailedBinding  8h (x3042 over 21h)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
  Normal  FailedBinding  4s (x2 over 19s)     persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


k describe pv nfs-pv-01
Name:            nfs-pv-01
Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    slow
Status:          Available
Claim:
Reclaim Policy:  Recycle
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        5Gi
Node Affinity:   <none>
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.1.108
    Path:      /hdd1/nfs_share/nfsvol01
    ReadOnly:  false
Events:        <none>

bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. `<type>` helper program

It could be the nfs client sw missing on the nodes (cp and all workers)
- sudo apt-get install nfs-common
Or it can be a missing mount options

  mountOptions:
    - hard
    - nfsvers=4.1

  Warning  FailedMount       25s (x8 over 89s)  kubelet            MountVolume.SetUp failed for volume "nfs-pv-02" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o hard,nfsvers=4.1 192.168.1.108:/hdd1/nfs_share/nfsvol02 /var/lib/kubelet/pods/b475fc5c-0d1e-483b-97df-de65feab09c9/volumes/kubernetes.io~nfs/nfs-pv-02
Output: mount: /var/lib/kubelet/pods/b475fc5c-0d1e-483b-97df-de65feab09c9/volumes/kubernetes.io~nfs/nfs-pv-02: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
cadm@cluster1:~/statfulset_k8s$

  Warning  FailedMount  36s (x60 over 106m)  kubelet  MountVolume.SetUp failed for volume "pvvol-1" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o hard,nfsvers=4.1 192.168.1.102:/hdd1/nfs_data /var/lib/kubelet/pods/303b4ead-4dc7-45df-ac09-511da72b096d/volumes/kubernetes.io~nfs/pvvol-1
Output: mount: /var/lib/kubelet/pods/303b4ead-4dc7-45df-ac09-511da72b096d/volumes/kubernetes.io~nfs/pvvol-1: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

This is how the persisten volume YAML looks

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-one
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 200Mi
ansible@labclient:~/tmp$ ls
PVol.yaml  favorite  nfs-pod.yaml  nfs-pod.yaml.org  primary  pvc.yaml  simpleshell.yaml
ansible@labclient:~/tmp$ cat PVol.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvvol-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /hdd1/nfs_data
    server: 192.168.1.102
    readOnly: false
  mountOptions:
  - hard
  - nfsvers=4.1