U1.29 Ubuntu Quick Start (QS): Kubespray on premises Installation Errors - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki
- read the article first Installing Kubernetes on premises with Kubespray.
our new hosts.yaml
Click to show the file
all:
hosts:
u2004d01:
ansible_host: 192.168.100.2
ip: 192.168.100.2
access_ip: 192.168.100.2
u2004d02:
ansible_host: 192.168.100.9
ip: 192.168.100.9
access_ip: 192.168.100.9
u2004d03:
ansible_host: 192.168.100.13
ip: 192.168.100.13
access_ip: 192.168.100.13
children:
kube_control_plane:
hosts:
u2004d01:
u2004d02:
kube_node:
hosts:
u2004d01:
u2004d02:
u2004d03:
etcd:
hosts:
u2004d01:
u2004d02:
u2004d03:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}
ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root -K cluster.yml
- We had first fatal message:
ASK [kubernetes/node : Modprobe nf_conntrack_ipv4] ******************************************
fatal: [u2004d02]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [u2004d01]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [u2004d03]: FAILED! => {"changed": false, "msg": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "name": "nf_conntrack_ipv4", "params": "", "rc": 1, "state": "present", "stderr": "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic\n", "stderr_lines": ["modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/5.11.0-40-generic"], "stdout": "", "stdout_lines": []}
...ignoring
Friday 26 November 2021 15:08:37 +0300 (0:00:00.648) 0:17:01.571 *******
- We had second fatal message:
ASK [kubernetes-apps/ansible : Kubernetes Apps | Register coredns deployment annotation `createdby`] ***
fatal: [u2004d01]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/kubectl get deploy -n kube-system coredns -o jsonpath='{ .spec.template.metadata.annotations.createdby }'", "delta": "0:00:01.279330", "end": "2021-11-26 15:14:30.021678", "msg": "non-zero return code", "rc": 1, "start": "2021-11-26 15:14:28.742348", "stderr": "Error from server (NotFound): deployments.apps \"coredns\" not found", "stderr_lines": ["Error from server (NotFound): deployments.apps \"coredns\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
Friday 26 November 2021 15:14:30 +0300 (0:00:02.225) 0:22:53.892 *******
- We had at the end:
PLAY RECAP ***********************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
u2004d01 : ok=611 changed=131 unreachable=0 failed=0 skipped=1181 rescued=0 ignored=2
u2004d02 : ok=546 changed=117 unreachable=0 failed=0 skipped=1024 rescued=0 ignored=1
u2004d03 : ok=466 changed=99 unreachable=0 failed=0 skipped=707 rescued=0 ignored=1
Friday 26 November 2021 15:15:37 +0300 (0:00:00.106) 0:24:01.520 *******
- login into u2004d01 (or u2004d02, or u2004d03)
- we saw repeating messages
- it was at 22 November 2021.
-
But there are no such a messages at 26 November 2021.
- Installation was carried out on three new virtual machines that will be used for Kubernetes.
- New separate virtual machine was used to run the Kubespray scripts (on the second datacenter).
- The only difference: both control_plane machines were added to kube_node section on 26 November 2021.
- we saw repeating messages
IPVS: rr: UDP 10.223.0.3:53 - no destination available
Click to show the picture
- read the article IPVS: rr: TCP *** - no destination available
- read the article IPVS - no destination available
- google - no destination available
- login into u2004d01 (or u2004d02, or u2004d03)
- got the same
IPVS: rr: UDP 10.223.0.3:53 - no destination available
Click to show the picture
- for the machine u2004d01 (or u2004d02)
sudo kubectl get nodes
- here is a result
NAME STATUS ROLES AGE VERSION
u2004d01 Ready control-plane,master 38m v1.22.3
u2004d02 Ready control-plane,master 37m v1.22.3
u2004d03 Ready <none> 35m v1.22.3
- for the machine u2004d01 (or u2004d02)
- run the command
sudo kubectl get pods -o wide
- here is a result
No resources found in default namespace.
- for the machine u2004d01 (or u2004d02)
- run the command
sudo kubectl get pods --all-namespaces -o wide
Click to show the result
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-684bcfdc59-wfvnf 1/1 Running 4 (3h23m ago) 5h43m 192.168.100.13 u2004d03 <none> <none>
kube-system calico-node-8r5nk 1/1 Running 1 (3h40m ago) 5h45m 192.168.100.9 u2004d02 <none> <none>
kube-system calico-node-v4mzw 1/1 Running 1 (3h42m ago) 5h45m 192.168.100.2 u2004d01 <none> <none>
kube-system calico-node-xldnd 1/1 Running 3 (3h23m ago) 5h45m 192.168.100.13 u2004d03 <none> <none>
kube-system coredns-8474476ff8-2xxcl 0/1 Pending 0 5h42m <none> <none> <none> <none>
kube-system coredns-8474476ff8-gz277 1/1 Running 2 (3h26m ago) 5h43m 10.233.73.7 u2004d03 <none> <none>
kube-system dns-autoscaler-5ffdc7f89d-64bpx 1/1 Running 2 (3h26m ago) 5h43m 10.233.73.6 u2004d03 <none> <none>
kube-system kube-apiserver-u2004d01 1/1 Running 2 (3h40m ago) 5h48m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-apiserver-u2004d02 1/1 Running 1 (3h40m ago) 5h47m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-controller-manager-u2004d01 1/1 Running 2 (3h42m ago) 5h48m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-controller-manager-u2004d02 1/1 Running 2 (3h40m ago) 5h47m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-proxy-77l2n 1/1 Running 1 (3h40m ago) 5h45m 192.168.100.9 u2004d02 <none> <none>
kube-system kube-proxy-mlfnp 1/1 Running 1 (3h42m ago) 5h45m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-proxy-p7cvm 1/1 Running 2 (3h26m ago) 5h45m 192.168.100.13 u2004d03 <none> <none>
kube-system kube-scheduler-u2004d01 1/1 Running 2 (3h42m ago) 5h48m 192.168.100.2 u2004d01 <none> <none>
kube-system kube-scheduler-u2004d02 1/1 Running 2 (3h40m ago) 5h47m 192.168.100.9 u2004d02 <none> <none>
kube-system nginx-proxy-u2004d03 1/1 Running 2 (3h26m ago) 5h45m 192.168.100.13 u2004d03 <none> <none>
kube-system nodelocaldns-6ptph 1/1 Running 2 (3h26m ago) 5h43m 192.168.100.13 u2004d03 <none> <none>
kube-system nodelocaldns-mgml8 1/1 Running 1 (3h40m ago) 5h43m 192.168.100.9 u2004d02 <none> <none>
kube-system nodelocaldns-p6dz9 1/1 Running 1 (3h42m ago) 5h43m 192.168.100.2 u2004d01 <none> <none>
- for the machine u2004d01 (or u2004d02)
- run the command
sudo kubectl describe node u2004d01
Click to show the result
Name: u2004d01
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=u2004d01
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.100.2/24
projectcalico.org/IPv4IPIPTunnelAddr: 10.233.72.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 26 Nov 2021 15:09:36 +0300
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: u2004d01
AcquireTime: <unset>
RenewTime: Fri, 26 Nov 2021 20:05:47 +0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 26 Nov 2021 17:19:26 +0300 Fri, 26 Nov 2021 17:19:26 +0300 CalicoIsUp Calico is running on this node
MemoryPressure False Fri, 26 Nov 2021 20:05:43 +0300 Fri, 26 Nov 2021 15:09:32 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 26 Nov 2021 20:05:43 +0300 Fri, 26 Nov 2021 17:18:48 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 26 Nov 2021 20:05:43 +0300 Fri, 26 Nov 2021 15:09:32 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 26 Nov 2021 20:05:43 +0300 Fri, 26 Nov 2021 15:13:49 +0300 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.100.2
Hostname: u2004d01
Capacity:
cpu: 1
ephemeral-storage: 12013044Ki
hugepages-2Mi: 0
memory: 1945660Ki
pods: 110
Allocatable:
cpu: 800m
ephemeral-storage: 11071221333
hugepages-2Mi: 0
memory: 1318972Ki
pods: 110
System Info:
Machine ID: 62506761d4d841688a78fef1b632975b
System UUID: 73b4ef04-ee90-2342-9303-b2b1c24fad5f
Boot ID: 2ce39e41-d0e1-4b54-864d-e36cc4d00c66
Kernel Version: 5.11.0-40-generic
OS Image: Ubuntu 20.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.4.11
Kubelet Version: v1.22.3
Kube-Proxy Version: v1.22.3
PodCIDR: 10.233.64.0/24
PodCIDRs: 10.233.64.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-v4mzw 150m (18%) 300m (37%) 64M (4%) 500M (37%) 4h52m
kube-system kube-apiserver-u2004d01 250m (31%) 0 (0%) 0 (0%) 0 (0%) 4h55m
kube-system kube-controller-manager-u2004d01 200m (25%) 0 (0%) 0 (0%) 0 (0%) 4h56m
kube-system kube-proxy-mlfnp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4h53m
kube-system kube-scheduler-u2004d01 100m (12%) 0 (0%) 0 (0%) 0 (0%) 4h56m
kube-system nodelocaldns-p6dz9 100m (12%) 0 (0%) 70Mi (5%) 170Mi (13%) 4h50m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 800m (100%) 300m (37%)
memory 137400320 (10%) 678257920 (50%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FreeDiskSpaceFailed 3m24s (x24 over 118m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 586769203 bytes, but freed 0 bytes
- for the machine u2004d01 (or u2004d02)
- run the command
sudo kubectl describe node u2004d03
Click to show the result
Name: u2004d03
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=u2004d03
kubernetes.io/os=linux
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.100.13/24
projectcalico.org/IPv4IPIPTunnelAddr: 10.233.73.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 26 Nov 2021 15:12:20 +0300
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: u2004d03
AcquireTime: <unset>
RenewTime: Fri, 26 Nov 2021 20:13:17 +0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 26 Nov 2021 17:35:16 +0300 Fri, 26 Nov 2021 17:35:16 +0300 CalicoIsUp Calico is running on this node
MemoryPressure False Fri, 26 Nov 2021 20:13:20 +0300 Fri, 26 Nov 2021 17:33:55 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 26 Nov 2021 20:13:20 +0300 Fri, 26 Nov 2021 17:33:55 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 26 Nov 2021 20:13:20 +0300 Fri, 26 Nov 2021 17:33:55 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 26 Nov 2021 20:13:20 +0300 Fri, 26 Nov 2021 17:33:55 +0300 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.100.13
Hostname: u2004d03
Capacity:
cpu: 1
ephemeral-storage: 12013044Ki
hugepages-2Mi: 0
memory: 1945660Ki
pods: 110
Allocatable:
cpu: 900m
ephemeral-storage: 11071221333
hugepages-2Mi: 0
memory: 1581116Ki
pods: 110
System Info:
Machine ID: ced833383fd2411b8c6bda3f1bc4de09
System UUID: 3c241d69-3702-354b-b20d-ca5f75e6cc0d
Boot ID: 06b5192c-612c-490e-b753-c56b7b4fa9d7
Kernel Version: 5.11.0-40-generic
OS Image: Ubuntu 20.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.4.11
Kubelet Version: v1.22.3
Kube-Proxy Version: v1.22.3
PodCIDR: 10.233.66.0/24
PodCIDRs: 10.233.66.0/24
Non-terminated Pods: (7 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-kube-controllers-684bcfdc59-wfvnf 30m (3%) 1 (111%) 64M (3%) 256M (15%) 4h59m
kube-system calico-node-xldnd 150m (16%) 300m (33%) 64M (3%) 500M (30%) 5h
kube-system coredns-8474476ff8-gz277 100m (11%) 0 (0%) 70Mi (4%) 170Mi (11%) 4h58m
kube-system dns-autoscaler-5ffdc7f89d-64bpx 20m (2%) 0 (0%) 10Mi (0%) 0 (0%) 4h58m
kube-system kube-proxy-p7cvm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5h
kube-system nginx-proxy-u2004d03 25m (2%) 0 (0%) 32M (1%) 0 (0%) 5h1m
kube-system nodelocaldns-6ptph 100m (11%) 0 (0%) 70Mi (4%) 170Mi (11%) 4h58m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 425m (47%) 1300m (144%)
memory 317286400 (19%) 1112515840 (68%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FreeDiskSpaceFailed 59m kubelet failed to garbage collect required amount of images. Wanted to free 501625651 bytes, but freed 0 bytes
Warning ImageGCFailed 59m kubelet failed to garbage collect required amount of images. Wanted to free 501625651 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 54m kubelet failed to garbage collect required amount of images. Wanted to free 501818163 bytes, but freed 0 bytes
Warning ImageGCFailed 54m kubelet failed to garbage collect required amount of images. Wanted to free 501818163 bytes, but freed 0 bytes
Warning ImageGCFailed 49m kubelet failed to garbage collect required amount of images. Wanted to free 510378803 bytes, but freed 0 bytes
Warning ImageGCFailed 34m (x3 over 44m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 510939955 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 4m43s (x10 over 49m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 512004915 bytes, but freed 0 bytes