U1.33 Ubuntu Quick Start (QS): Kubernetes on premises and Docker and Kubespray. - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki
- currently k8s-cluster.yml declares
- container_manager: containerd
- to install Kubernetes with docker we should set
- container_manager: docker
-
Pre-installed DHCP in the virtual environment (for example, a hardware implementation of a DHCP server in a modem)
-
Deploy four Ubuntu 20.04 TLS virtual machines. Consult the articles U1.01 and U1.02
- Device name = u2004d01, ip = 192.168.100.2
- Device name = u2004d02, ip = 192.168.100.9
- Device name = u2004d03, ip = 192.168.100.13
- Device name = UbuntuAnsible, ip = it does not matter
-
For each machine u200401, u200402, u200403, UbuntuAnsible
- run the command python3 --version
- In our case it returns : Python 3.8.10
- run the command python3 --version
-
For each machine u200401, u200402, u200403, UbuntuAnsible
- we have sudo-enabled user = yury with identical password for each machine
- For each machine u200401, u200402, u200403
- run the command
sudo apt install openssh-server
sudo systemctl set-default multi-user
sudo reboot
- run the commands (no sudo prefix for the first four commands and for git-command)
- Note: cp -rfp inventory... command below creates a new folder named cluster
ssh-keygen
ssh-copy-id 192.168.100.2
ssh-copy-id 192.168.100.9
ssh-copy-id 192.168.100.13
sudo apt-get install python3-pip
sudo pip3 install --upgrade pip
sudo apt-get install git
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
sudo pip3 install -r requirements.txt
cp -rfp inventory/sample inventory/cluster
nano inventory/cluster/group_vars/k8s_cluster/k8s-cluster.yml
-
in the file k8s-cluster.yml modify definition of container_manager-var
- set container_manager: docker and save the file
-
populate inventory/cluster/hosts.yaml-file with IP-addresses of our machines
declare -a IPS=(192.168.100.2 192.168.100.9 192.168.100.13)
CONFIG_FILE=inventory/cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
- modify node names in the inventory/cluster/hosts.yaml-file.
nano inventory/cluster/hosts.yaml
Click to show hosts.yaml
all:
hosts:
u2004d01:
ansible_host: 192.168.100.2
ip: 192.168.100.2
access_ip: 192.168.100.2
u2004d02:
ansible_host: 192.168.100.9
ip: 192.168.100.9
access_ip: 192.168.100.9
u2004d03:
ansible_host: 192.168.100.13
ip: 192.168.100.13
access_ip: 192.168.100.13
children:
kube_control_plane:
hosts:
u2004d01:
u2004d02:
kube_node:
hosts:
u2004d01:
u2004d02:
u2004d03:
etcd:
hosts:
u2004d01:
u2004d02:
u2004d03:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}
- read the article Kubernetes Release History
- in our case it's 1.22.3
- reset kube_version: v1.22.3
nano inventory/cluster/group_vars/k8s_cluster/k8s-cluster.yml
- start installation (no sudo prefix for the command)
ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root -K cluster.yml
Click to show the first failure
TASK [kubernetes/node : Modprobe nf_conntrack_ipv4] ****************************
fatal: [u2004d01]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [u2004d02]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [u2004d03]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
Sunday 28 November 2021 21:02:41 +0300 (0:00:00.634) 0:18:50.778 *******
Click to show the second failure
TASK [kubernetes-apps/ansible : Kubernetes Apps | Register coredns deployment annotation `createdby`] ***
fatal: [u2004d01]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/kubectl get deploy -n kube-system coredns -o jsonpath='{ .spec.template.metadata.annotations.createdby }'", "delta": "0:00:00.203512", "end": "2021-11-28 21:07:49.647198", "msg": "non-zero return code", "rc": 1, "start": "2021-11-28 21:07:49.443686", "stderr": "Error from server (NotFound): deployments.apps \"coredns\" not found", "stderr_lines": ["Error from server (NotFound): deployments.apps \"coredns\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
Sunday 28 November 2021 21:07:49 +0300 (0:00:00.835) 0:23:58.733 *******
Click to view the final report
PLAY RECAP *********************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
u2004d01 : ok=582 changed=126 unreachable=0 failed=0 skipped=1157 rescued=0 ignored=2
u2004d02 : ok=518 changed=113 unreachable=0 failed=0 skipped=1011 rescued=0 ignored=1
u2004d03 : ok=438 changed=95 unreachable=0 failed=0 skipped=694 rescued=0 ignored=1
Sunday 28 November 2021 21:08:58 +0300 (0:00:00.121) 0:25:07.708 *******
- run the command
sudo kubectl sudo kubectl get nodes -o wide
Click to view the responce
yury@u2004d01:~$ sudo kubectl get nodes -o wide
[sudo] password for yury:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
u2004d01 Ready control-plane,master 11m v1.22.3 192.168.100.2 <none> Ubuntu 20.04.3 LTS 5.11.0-40-generic docker://20.10.9
u2004d02 Ready control-plane,master 10m v1.22.3 192.168.100.9 <none> Ubuntu 20.04.3 LTS 5.11.0-40-generic docker://20.10.9
u2004d03 Ready <none> 8m39s v1.22.3 192.168.100.13 <none> Ubuntu 20.04.3 LTS 5.11.0-40-generic docker://20.10.9
- run the command
- our nodelocaldns-7z2nr-pod has Pending-status
sudo kubectl get pods --all-namespaces
Click to view the responce
yury@u2004d01:~$ sudo kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-684bcfdc59-c847k 1/1 Running 0 11m
kube-system calico-node-4sfp9 1/1 Running 0 11m
kube-system calico-node-brngh 1/1 Running 0 11m
kube-system calico-node-mkjmf 1/1 Running 0 11m
kube-system coredns-8474476ff8-jzbd4 1/1 Running 0 10m
kube-system coredns-8474476ff8-vxgfx 1/1 Running 0 10m
kube-system dns-autoscaler-5ffdc7f89d-k2jxg 1/1 Running 0 10m
kube-system kube-apiserver-u2004d01 1/1 Running 0 14m
kube-system kube-apiserver-u2004d02 1/1 Running 0 14m
kube-system kube-controller-manager-u2004d01 1/1 Running 1 15m
kube-system kube-controller-manager-u2004d02 1/1 Running 1 14m
kube-system kube-proxy-5zs4v 1/1 Running 0 12m
kube-system kube-proxy-696sc 1/1 Running 0 12m
kube-system kube-proxy-wznq9 1/1 Running 0 12m
kube-system kube-scheduler-u2004d01 1/1 Running 1 15m
kube-system kube-scheduler-u2004d02 1/1 Running 1 14m
kube-system nginx-proxy-u2004d03 1/1 Running 0 11m
kube-system nodelocaldns-7z2nr 0/1 Pending 0 10m
kube-system nodelocaldns-dbn98 1/1 Running 0 10m
kube-system nodelocaldns-l2zh4 1/1 Running 0 10m
- run the command
- our nodelocaldns-7z2nr-pod has Pending-status
sudo kubectl describe pod -n kube-system nodelocaldns-7z2nr
Click to view the Events of the responce
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 13m default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector.
Warning FailedScheduling 12m default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector.
- step 1: for u2004d01, u2004d02, u2004d03
sudo poweroff
-
step 2: with hyper-v define two virtual CPU for each machine
- u2004d01 has 2 vcpu
- u2004d02 has 2 vcpu
- u2004d03 has 2 vcpu
-
step 3: start machines
Note: to tell the truth, restarint all machines without adding vcpu gives the same result (it was tested)
sudo kubectl sudo kubectl get nodes -o wide
The response is as shown above. All nodes are ready.
sudo kubectl get pods --all-namespaces -o wide
Click to view the Events of the responce
yury@u2004d01:~$ sudo kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-684bcfdc59-c847k 1/1 Running 2 (9m19s ago) 37m 192.168.100.13 u2004d03 <none> <none>
kube-system calico-node-4sfp9 1/1 Running 1 (15m ago) 38m 192.168.100.9 u2004d02 <none> <none>
kube-system calico-node-brngh 1/1 Running 1 (16m ago) 38m 192.168.100.2 u2004d01 <none> <none>
kube-system calico-node-mkjmf 1/1 Running 1 (14m ago) 38m 192.168.100.13 u2004d03 <none> <none>
kube-system coredns-8474476ff8-jzbd4 1/1 Running 1 (15m ago) 37m 10.233.84.2 u2004d02 <none> <none>
kube-system coredns-8474476ff8-vxgfx 1/1 Running 1 (14m ago) 36m 10.233.72.4 u2004d03 <none> <none>
kube-system dns-autoscaler-5ffdc7f89d-k2jxg 1/1 Running 1 (15m ago) 37m 10.233.72.3 u2004d03 <none> <none>
kube-system kube-apiserver-u2004d01 1/1 Running 1 (15m ago) 41m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-apiserver-u2004d02 1/1 Running 1 (15m ago) 41m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-controller-manager-u2004d01 1/1 Running 2 (16m ago) 41m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-controller-manager-u2004d02 1/1 Running 2 (15m ago) 41m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-proxy-5zs4v 1/1 Running 1 (16m ago) 39m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-proxy-696sc 1/1 Running 1 (15m ago) 39m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-proxy-wznq9 1/1 Running 1 (14m ago) 39m 192.168.100.13 u2004d03 <none> <none>
kube-system kube-scheduler-u2004d01 1/1 Running 2 (16m ago) 41m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-scheduler-u2004d02 1/1 Running 2 (15m ago) 41m 192.168.100.9 u2004d02 <none> <none>
kube-system nginx-proxy-u2004d03 1/1 Running 1 (14m ago) 38m 192.168.100.13 u2004d03 <none> <none>
kube-system nodelocaldns-7z2nr 1/1 Running 0 37m 192.168.100.9 u2004d02 <none> <none>
kube-system nodelocaldns-dbn98 1/1 Running 1 (16m ago) 37m 192.168.100.2 u2004d01 <none> <none>
kube-system nodelocaldns-l2zh4 1/1 Running 1 (14m ago) 37m 192.168.100.13 u2004d03 <none> <none>
sudo kubectl describe node u2004d01
Click to view the Events of the responce
yury@u2004d01:~$ sudo kubectl describe node u2004d01
Name: u2004d01
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=u2004d01
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.100.2/24
projectcalico.org/IPv4IPIPTunnelAddr: 10.233.73.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 28 Nov 2021 21:03:32 +0300
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: u2004d01
AcquireTime: <unset>
RenewTime: Sun, 28 Nov 2021 21:49:37 +0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sun, 28 Nov 2021 21:35:59 +0300 Sun, 28 Nov 2021 21:35:59 +0300 CalicoIsUp Calico is running on this node
MemoryPressure False Sun, 28 Nov 2021 21:49:38 +0300 Sun, 28 Nov 2021 21:35:21 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 28 Nov 2021 21:49:38 +0300 Sun, 28 Nov 2021 21:35:21 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 28 Nov 2021 21:49:38 +0300 Sun, 28 Nov 2021 21:35:21 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 28 Nov 2021 21:49:38 +0300 Sun, 28 Nov 2021 21:35:21 +0300 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.100.2
Hostname: u2004d01
Capacity:
cpu: 2
ephemeral-storage: 12013044Ki
hugepages-2Mi: 0
memory: 1945700Ki
pods: 110
Allocatable:
cpu: 1800m
ephemeral-storage: 11071221333
hugepages-2Mi: 0
memory: 1319012Ki
pods: 110
System Info:
Machine ID: 62506761d4d841688a78fef1b632975b
System UUID: 73b4ef04-ee90-2342-9303-b2b1c24fad5f
Boot ID: 70701960-f754-480a-a37d-d0d9ff65295b
Kernel Version: 5.11.0-40-generic
OS Image: Ubuntu 20.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.9
Kubelet Version: v1.22.3
Kube-Proxy Version: v1.22.3
PodCIDR: 10.233.64.0/24
PodCIDRs: 10.233.64.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-brngh 150m (8%) 300m (16%) 64M (4%) 500M (37%) 42m
kube-system kube-apiserver-u2004d01 250m (13%) 0 (0%) 0 (0%) 0 (0%) 45m
kube-system kube-controller-manager-u2004d01 200m (11%) 0 (0%) 0 (0%) 0 (0%) 46m
kube-system kube-proxy-5zs4v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 43m
kube-system kube-scheduler-u2004d01 100m (5%) 0 (0%) 0 (0%) 0 (0%) 46m
kube-system nodelocaldns-dbn98 100m (5%) 0 (0%) 70Mi (5%) 170Mi (13%) 41m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 800m (44%) 300m (16%)
memory 137400320 (10%) 678257920 (50%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 45m kube-proxy
Normal Starting 14m kube-proxy
Normal Starting 43m kube-proxy
Normal NodeHasSufficientMemory 45m kubelet Node u2004d01 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 45m kubelet Node u2004d01 status is now: NodeHasSufficientPID
Normal Starting 45m kubelet Starting kubelet.
Normal NodeHasNoDiskPressure 45m (x2 over 45m) kubelet Node u2004d01 status is now: NodeHasNoDiskPressure
Normal NodeAllocatableEnforced 45m kubelet Updated Node Allocatable limit across pods
Normal Starting 44m kubelet Starting kubelet.
Normal NodeHasNoDiskPressure 44m kubelet Node u2004d01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 44m kubelet Node u2004d01 status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 44m kubelet Node u2004d01 status is now: NodeHasSufficientMemory
Normal NodeAllocatableEnforced 44m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 42m kubelet Node u2004d01 status is now: NodeReady
Normal NodeHasDiskPressure 41m kubelet Node u2004d01 status is now: NodeHasDiskPressure
Warning FreeDiskSpaceFailed 39m kubelet failed to garbage collect required amount of images. Wanted to free 723755827 bytes, but freed 0 bytes
Warning ImageGCFailed 39m kubelet failed to garbage collect required amount of images. Wanted to free 723755827 bytes, but freed 0 bytes
Warning EvictionThresholdMet 38m (x17 over 41m) kubelet Attempting to reclaim ephemeral-storage
Warning FreeDiskSpaceFailed 34m kubelet failed to garbage collect required amount of images. Wanted to free 725459763 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 29m kubelet failed to garbage collect required amount of images. Wanted to free 725582643 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 24m kubelet failed to garbage collect required amount of images. Wanted to free 725693235 bytes, but freed 0 bytes
Normal Starting 15m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 15m (x8 over 15m) kubelet Node u2004d01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 15m (x8 over 15m) kubelet Node u2004d01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 15m (x7 over 15m) kubelet Node u2004d01 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 15m kubelet Updated Node Allocatable limit across pods
- we have a new problem with garbage collection
Warning FreeDiskSpaceFailed 34m kubelet failed to garbage collect required amount of images. Wanted to free 725459763 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 29m kubelet failed to garbage collect required amount of images. Wanted to free 725582643 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 24m kubelet failed to garbage collect required amount of images. Wanted to free 725693235 bytes, but freed 0 bytes
- step 1: for u2004d01, u2004d02, u2004d03
- run the command
df -h
Click to view the Events of the responce
yury@u2004d01:~$ df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sda1 12G 9,5G 2,0G 83% /
...
- We used 83% of the disk space
- Let's assume that this is the reason for the warnings.
- Follow the instructions in the article U1.32 Ubuntu Quick Start (QS): Expand Ubuntu disk (Warning FreeDiskSpaceFailed)
- we set new disk size = 20Gb
- running fd -h gives the response
yury@u2004d01:~$ df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sda1 20G 9,5G 9,7G 50% /
...
- login into u2004d01
- run the command for u2004d01
sudo kubectl describe node u2004d01
- here is a response
- the events section has disappeared
yury@u2004d01:~$ sudo kubectl describe node u2004d01
...
Normal NodeHasSufficientMemory 37m (x8 over 37m) kubelet Node u2004d01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 37m (x8 over 37m) kubelet Node u2004d01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 37m (x7 over 37m) kubelet Node u2004d01 status is now: NodeHasSufficientPID
...
- repeat the command to make sure the report is the same
sudo kubectl describe node u2004d02
sudo kubectl describe node u2004d03