How to Scale your Cluster - gpillon/k4all GitHub Wiki

K4all Cluster Documentation

This documentation covers the concepts, configuration, and installation process for K4all clusters with more than one node. The project supports multiple High Availability (HA) modes and provides a set of scripts to bootstrap the cluster and join new nodes.

This page is for the bootc line (v2.x). On v1 (Fedora CoreOS) the settings were in config.json, see v1 (Fedora CoreOS) at the end.


Overview

K4all uses a bootstrap node (a control plane node) to start the cluster. After bootstrapping, additional nodes (either worker or control plane nodes) can join the cluster using dedicated join scripts.

High Availability (HA) Modes for Control Plane

During installation you can choose one of these HA modes:

  • kubevip Sets up a virtual IP for the API server using kubevip.

    • From v2.4.0, kube-vip runs on every control plane. join_cluster.sh writes it right after a control plane joins, and k4all-upgrade writes it again after every upgrade hop (k4all-upgrade kube-vip-manifest). Up to v2.3.0 only the bootstrap node runs it.
    • From v2.4.0 kube-vip also announces Traefik's dedicated IP: see MetalLB Configuration.
    • On v2.0 – v2.2.3, read Known issue: kube-vip on HA clusters (v2.0–v2.2.3).
  • keepalived Uses keepalived to manage virtual IPs.

    Note: This mode is not recommended and may be removed in a future release.

  • External Load Balancer Use your own load balancer: keep the HA mode at none and set the API endpoint by hand ("Custom API endpoint" in the installer, spec.cluster.customApiEndPoint in the config).


Installation Process

  1. Bootstrapping the Cluster

    • Install the first node from the k4all-bootstrap ISO (unattended, default settings: no HA) or from the k4all-attended ISO.
    • The attended ISO shows the standard Anaconda screens plus a K4All screen. There you pick the node role, the CNI and the High Availability mode (None, Keepalived, KubeVIP). For HA, also fill in Control Plane VIP (a free IP on the same subnet) and VIP Subnet Size.
    • With a kickstart, the same settings are options of the K4All addon, for example:
      %addon com_k4all_installer --role=bootstrap --ha=kubevip --api-control-endpoint=<VIP> --api-control-endpoint-subnet=24
      %end
      
    • For kube-vip, set the HA mode on the bootstrap node only. The other control planes keep ha.type: none in their own config, and kube-vip takes its settings from the cluster's ClusterConfig.
  2. Downloading Node Images

    • Each release ships four ISOs, for x86_64 and aarch64:
      • k4all-bootstrap: the first node (unattended)
      • k4all-control: an additional control plane node (unattended)
      • k4all-worker: a worker node (unattended)
      • k4all-attended: an interactive install, for any role
    • How to download and unpack them: see Which ISO in the README.
  3. Generating Join Commands

    • Once the bootstrap node is running, use the generate_join.sh script to generate a join command for additional nodes.
    • The script supports two modes:
      • worker: for adding worker nodes.
      • control: for adding additional control plane nodes.
    • By default, the join command is encoded in base64. Use --debug to print it in clear text, or --unattended to print only the join command.
  4. Joining New Nodes

    • Install the new node from the k4all-control or k4all-worker ISO (or the attended ISO, with that role).
    • On the new node, run join_cluster.sh (different for worker and control nodes) with the join command as the argument.
    • The script decodes the join command, executes it, and sets up the kubeconfig. On control nodes it also removes the scheduling taint and, on HA kube-vip clusters, writes kube-vip.

Configuration File: /etc/k4all-config.yaml

On v2.x the node settings are a ClusterConfig resource in /etc/k4all-config.yaml. The installer writes it from the K4All screen or the kickstart, together with /etc/node-type (the node role).

The HA part looks like this:

apiVersion: k4all.magesgate.com/v1alpha1
kind: ClusterConfig
metadata:
  name: k4all-cluster-config
spec:
  cluster:
    apiEndPointUseHostName: false
    customApiEndPoint: ""            # external load balancer: its address
    ha:
      interface: auto                # network interface for the VIP ("auto": the default route)
      type: none                     # none | keepalived | kubevip
      apiControlEndpoint: ""         # the VIP
      apiControlEndpointSubnetSize: ""

The full schema, with its defaults, is bootc/overlay/etc/k4all-config.yaml.default in the repo. The installer options are in the K4All Anaconda addon README.


Scripts Overview

All scripts are in the system PATH for easy access.

generate_join.sh

This script generates a join command that is used by new nodes to join the cluster. Run it on the bootstrap node.

  • Usage:

    generate_join.sh [worker|control] [--debug|--unattended]
  • Parameters:

    • worker or control: Specifies whether the join command is for a worker node or a control plane node.
    • --debug: (Optional) Outputs the join command in clear text instead of base64 encoding it.
    • --unattended: (Optional) Outputs only the join command, nothing else.
  • Functionality:

    • Generates a join token using kubeadm token create.
    • For control plane nodes, it uploads certificates and appends the certificate key to the join command.
    • By default, the join command is encoded in base64, and the script prints the join_cluster.sh command to run.
    • For worker nodes, the script also waits for the new node to show up in the cluster, then labels it as a worker and deletes the join token.

Example:

# Generate a join command for a worker node (encoded)
generate_join.sh worker

# Generate a join command for a control plane node (clear text)
generate_join.sh control --debug

join_cluster.sh (Worker Version)

This script is executed on a new worker node to join the existing Kubernetes cluster.

  • Usage:

    join_cluster.sh '<base64_join_command>'
  • Process:

    • Decodes the provided join command from base64 and runs it.
    • Sets up kubeconfig for the root and core users (with kubelet.conf).
    • Verifies the join by reading its own Node with kubelet.conf. A worker may read only its own Node, so kubectl get nodes doesn't work there.

Source: overrides/worker/join_cluster.sh.


join_cluster.sh (Control Plane Version)

This variant is used on a new control plane node.

  • Usage:

    join_cluster.sh '<base64_join_command>'
  • Process:

    • Decodes and executes the join command.
    • Sets up kubeconfig for both root and core users.
    • Verifies that the node has joined the cluster.
    • Removes the default taint from the control plane node, allowing workloads to be scheduled if needed.
    • On an HA kube-vip cluster, after a join that worked, writes kube-vip's static pod with k4all-upgrade kube-vip-manifest --write. If that fails, it prints a warning: run that command again by hand.

Source: overrides/control/join_cluster.sh.


Upgrading a multi-node cluster

Upgrades go one node at a time: the bootstrap node, then the other control planes, then the workers. Every hop reboots the node. The whole story is in Upgrading k4all (v2.x) and in docs/upgrades.md, Multi-node clusters.

  • From v2.4.0, use a ClusterUpgrade, from the Headlamp plugin (k4all > Upgrades). The k4all-operator takes the nodes in order, one at a time. It cordons and drains each one, and uncordons it once it is back. Set mode: managed in /etc/k4all-upgrade.yaml on every node.
  • Before that (the bridge, v2.3.0, and v2.4.0 itself), go node by node from the command line, with mode: manual on every node. Control planes drain themselves. A worker can't: drain it from a control plane, and its hop starts once it is cordoned:
    kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
    # after the hop, once the node is back:
    kubectl uncordon <node>
  • Two control planes are not real HA. etcd needs more than half of its members. With two control planes it needs both, so every reboot of a control plane takes etcd, and the API, down until that node is back, also through the VIP. Workloads keep running, but nothing can change them meanwhile. For real HA use three control planes (or stay with one).

To remove a node, see Removing a Node.


Summary

  • Bootstrap Node: The first node in the cluster, installed from the bootstrap or attended ISO, and responsible for generating join tokens.

  • Join Process: Use generate_join.sh on the bootstrap node to generate a join command. Then, run join_cluster.sh on new nodes (installed from the control or worker ISO) to add them to the cluster.

  • HA Modes: Choose your HA mode (kubevip, keepalived, or external load balancer) at install time, on the bootstrap node. From v2.4.0 kube-vip runs on every control plane.

  • Upgrades: One node at a time. From v2.4.0 a ClusterUpgrade does the cordon and drain for you. Use three control planes for real HA.

By following these steps and using the provided scripts, you can easily deploy and scale your Kubernetes cluster with K4all.

Happy clustering!


v1 (Fedora CoreOS)

On v1 the settings were in config.json. You could edit it during the first 10 seconds of the installation (press 1 for Nano or 2 for Vi). The HA mode was the "node.ha.type" field ("kubevip", "keepalived" or "none"), and there were separate bootstrap, control plane and worker images. The v1 code is in the v1.x branch.

⚠️ **GitHub.com Fallback** ⚠️