Moving from ingress nginx to Traefik (v2.4.0) - gpillon/k4all GitHub Wiki

Moving from ingress-nginx to Traefik (v2.4.0)

From v2.4.0 the ingress controller of k4all is Traefik. It replaces ingress-nginx, whose last release is tested only up to Kubernetes 1.35.

The full details are in docs/upgrades.md, Ingress: Traefik.

What happens during the upgrade

  • During the v2.4.0 hop, the k4all-operator uninstalls ingress-nginx, then installs Traefik on the same LoadBalancer IP (or the same host ports).
  • The routes are down in between. Traefik comes after the first upgrade step of the CNI and of cert-manager, and after MetalLB, metrics-server, nmstate and TopoLVM. Plan for some minutes without routes.
  • The Traefik image ships with the node image, so Traefik doesn't wait for a pull.
  • From v2.6.0 the release manifest doesn't list ingress-nginx any more. Every cluster went through v2.4.0 (a mandatory stop), which uninstalled it.
  • v2.4.0 leaves an empty ingress-nginx namespace behind. From v2.6.1, k4all-operator v0.5.1 removes it once, when only the leftovers of the uninstall are in it; with your own objects inside, it stays.

k4all's own routes

  • The Headlamp dashboard and Argo CD routes use the traefik class.
  • Where ingress-nginx had force-ssl-redirect, they use the Middleware traefik/redirect-https, which comes with Traefik:
    traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-https@kubernetescrd
  • The dashboard route has a rule for dashboard.<FQDN> and one for dashboard.<IP>.nip.io. The IP is Traefik's dedicated IP, or the node's address when there is none.

Settings

  • The ingress settings are now spec.ingress.traefik in the ClusterConfig: dedicatedIP and isDefault. Clusters installed before v2.4.0 have spec.ingress.nginx, which moves to spec.ingress.traefik with the same values. You don't have to do anything:
    • on each node, k4all-migrate-config.sh moves it in /etc/k4all-config.yaml at the first boot of v2.4.0. The file as it was stays in /etc/k4all-config.yaml.ingress-nginx.migrated;
    • the k4all-operator moves it in the ClusterConfig during the v2.4.0 sync.
  • Kickstart: --ingress-traefik-ip, --no-ingress-traefik and --ingress-default=traefik|cilium. The old --ingress-nginx* options are still accepted.
  • On HA clusters with kube-vip, Traefik's dedicated IP is announced by kube-vip, not MetalLB. See MetalLB Configuration.

Your Ingresses of class nginx keep working

Your own Ingresses with ingressClassName: nginx and nginx.ingress.kubernetes.io/* annotations keep working. Traefik serves them through its Ingress-NGINX provider, ssl-passthrough included.

The nginx IngressClass stays for them, but it is never the default class. Still, it's best to move them to the traefik class.

Moving an Ingress to the traefik class

From the Headlamp plugin

k4all > Settings > Ingress lists the Ingresses still on the nginx class, and the Overview counts them. Switch to traefik sets the class and converts what has a Traefik equivalent:

  • nginx.ingress.kubernetes.io/force-ssl-redirect or ssl-redirect: "true" become the HTTPS redirect Middleware k4all provides;
  • nginx.ingress.kubernetes.io/backend-protocol: HTTPS moves to the backend Services (traefik.ingress.kubernetes.io/service.serversscheme).

Other nginx annotations stay, but stop having an effect. The review lists them and asks you to confirm.

You need the admin token or the k4all-editor role. See Headlamp plugin.

By hand

ingress-nginx Traefik
ingressClassName: nginx ingressClassName: traefik
nginx.ingress.kubernetes.io/force-ssl-redirect: "true" traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-https@kubernetescrd
nginx.ingress.kubernetes.io/backend-protocol: HTTPS traefik.ingress.kubernetes.io/service.serversscheme: https on the backend Service. Traefik verifies the backend certificate, so a self-signed one also needs a ServersTransport with insecureSkipVerify
nginx.ingress.kubernetes.io/ssl-passthrough: "true" an IngressRouteTCP with tls.passthrough: true

Ingresses from Helm or Argo CD

Change them where they come from (chart values, Git). If you don't, the next upgrade or sync puts the nginx class back. The plugin's review tells you when it sees one.

⚠️ **GitHub.com Fallback** ⚠️