Known issue kube vip on HA clusters - gpillon/k4all GitHub Wiki

Known issue: kube-vip on HA clusters (v2.0–v2.2.3)

This is a summary. The full details and the workaround are in issue #41. Background: docs/upgrades.md, kube-vip on HA clusters.

Who is affected

k4all v2.0 to v2.2.3 clusters installed with HA type kube-vip. Clusters without HA, or with keepalived, are not affected.

What happens

The k4all-operator (v0.1.x to v0.4.x) manages kube-vip as a static-pod component. When it applies it, it rewrites /etc/kubernetes/manifests/kube-vip.yaml with the image alone: no manager argument, no settings. kube-vip exits, and the API VIP is gone.

The operator applies it right after the install (on v2.2.x about two minutes after kubeadm init), and again every time kube-vip publishes a new release, because the version is latest. So most HA clusters on these releases have it.

Symptoms:

  • kubectl with /etc/kubernetes/admin.conf fails after about 15 seconds with connect: no route to host: that file points to the VIP;
  • through the bootstrap node's own IP (--server https://<bootstrap-ip>:6443) the nodes still look Ready;
  • sudo crictl ps -a --name kube-vip shows the container exiting;
  • sudo grep -c 'name: address' /etc/kubernetes/manifests/kube-vip.yaml prints 0.

What to do

On the bootstrap node:

  1. Tell the operator to leave kube-vip alone. If the VIP is down, add --server https://<bootstrap-ip>:6443:
    sudo kubectl --kubeconfig /etc/kubernetes/admin.conf patch releasemanifest k4all \
      --type merge -p '{"spec":{"components":{"kube-vip":{"management":"Unmanaged"}}}}'
  2. If the manifest was emptied, write it again. The template, and the values to fill in, are in #41 (step 2). The kubelet starts kube-vip again by itself, in a minute or two. Then check /etc/kubernetes/manifests/kube-vip.yaml on every other control plane: if it exists and has no address entry, delete it. v2.2.x installs kube-vip on the bootstrap node only.
  3. Check: ip -4 addr show <INTERFACE> lists the VIP, and curl -k https://<VIP>:6443/readyz answers ok.

Then take the bridge as usual: see Upgrading k4all (v2.x).

Fixed in

  • v2.2.4 (the bridge) sets kube-vip to Unmanaged by itself, so step 1 isn't needed after it. If the VIP is already down, the bridge stops before changing anything: put the manifest back first (step 2), then run the bridge again.
  • v2.4.0: kube-vip is Unmanaged in the release manifest. k4all writes kube-vip on every control plane (k4all-upgrade kube-vip-manifest), and the ingress IP is announced by kube-vip with its load balancer class. k4all-operator v0.5.0, pinned from v2.4.0, doesn't overwrite a static Pod manifest it didn't write.
⚠️ **GitHub.com Fallback** ⚠️