Known issue kube vip on HA clusters - gpillon/k4all GitHub Wiki
This is a summary. The full details and the workaround are in issue #41. Background: docs/upgrades.md, kube-vip on HA clusters.
k4all v2.0 to v2.2.3 clusters installed with HA type kube-vip. Clusters without HA, or with keepalived, are not affected.
The k4all-operator (v0.1.x to v0.4.x) manages kube-vip as a static-pod component. When it applies it, it rewrites /etc/kubernetes/manifests/kube-vip.yaml with the image alone: no manager argument, no settings. kube-vip exits, and the API VIP is gone.
The operator applies it right after the install (on v2.2.x about two minutes after kubeadm init), and again every time kube-vip publishes a new release, because the version is latest. So most HA clusters on these releases have it.
Symptoms:
-
kubectlwith/etc/kubernetes/admin.conffails after about 15 seconds withconnect: no route to host: that file points to the VIP; - through the bootstrap node's own IP (
--server https://<bootstrap-ip>:6443) the nodes still look Ready; -
sudo crictl ps -a --name kube-vipshows the container exiting; -
sudo grep -c 'name: address' /etc/kubernetes/manifests/kube-vip.yamlprints0.
On the bootstrap node:
- Tell the operator to leave kube-vip alone. If the VIP is down, add
--server https://<bootstrap-ip>:6443:sudo kubectl --kubeconfig /etc/kubernetes/admin.conf patch releasemanifest k4all \ --type merge -p '{"spec":{"components":{"kube-vip":{"management":"Unmanaged"}}}}' - If the manifest was emptied, write it again. The template, and the values to fill in, are in #41 (step 2). The kubelet starts kube-vip again by itself, in a minute or two. Then check
/etc/kubernetes/manifests/kube-vip.yamlon every other control plane: if it exists and has noaddressentry, delete it. v2.2.x installs kube-vip on the bootstrap node only. - Check:
ip -4 addr show <INTERFACE>lists the VIP, andcurl -k https://<VIP>:6443/readyzanswersok.
Then take the bridge as usual: see Upgrading k4all (v2.x).
-
v2.2.4 (the bridge) sets kube-vip to
Unmanagedby itself, so step 1 isn't needed after it. If the VIP is already down, the bridge stops before changing anything: put the manifest back first (step 2), then run the bridge again. -
v2.4.0: kube-vip is
Unmanagedin the release manifest. k4all writes kube-vip on every control plane (k4all-upgrade kube-vip-manifest), and the ingress IP is announced by kube-vip with its load balancer class. k4all-operator v0.5.0, pinned from v2.4.0, doesn't overwrite a static Pod manifest it didn't write.