How to Scale your Cluster - gpillon/k4all GitHub Wiki

K4all Cluster Documentation

This documentation covers the concepts, configuration, and installation process for the K4all. The project supports multiple High Availability (HA) modes and provides a set of scripts to bootstrap the cluster and join new nodes.

Overview

K4all uses a bootstrap node (a control plane node) to start the cluster. After bootstrapping, additional nodes (either worker or control plane nodes) can join the cluster using dedicated join scripts.

High Availability (HA) Modes for Control Plane

During installation you can choose one of these HA modes:

  • kubevip
    Sets up a virtual IP for the API server using kubevip.

  • keepalived
    Uses keepalived to manage virtual IPs.

    Note: This mode is not recommended and may be removed in a future release.

  • External Load Balancer
    Use your own load balancer by configuring the API endpoint manually in the configuration file.

Installation Process

  1. Bootstrapping the Cluster

    • Boot a node with the K4all image.
    • Within the first 10 seconds of the installation, you are prompted to edit the configuration file (config.json).
      • Press 1 to open with Nano or 2 to open with Vi.
    • Adjust the settings according to your environment. In particular, set the HA mode in the "node.ha.type" field:
      • Set to "kubevip" if you want to use kubevip (only on control plane nodes).
      • Set to "keepalived" (not recommended) or "none" if you plan to use an external load balancer.

  2. Downloading Node Images

    • Different images are available for:
      • Bootstrap nodes
      • Worker nodes
      • Control plane nodes
    • All images can be found in the release section of the repository. Make sure to download and deploy the correct image for each node type.

  3. Generating Join Commands

    • Once the bootstrap node is running, use the generate_join.sh script to generate a join command for additional nodes.
    • The script supports two modes:
      • worker: for adding worker nodes.
      • control: for adding additional control plane nodes.
    • By default, the join command is encoded in base64. Use the --debug flag to print it in clear text if needed.

  4. Joining New Nodes

    • On each new node, run the appropriate join_cluster.sh script (different for worker and control nodes) with the join command as the argument.
    • The script decodes the join command, executes it, and performs necessary configurations such as setting up kubeconfig and (for control nodes) removing scheduling taints.

Configuration File: config.json

The config.json file is read during boot to configure networking, disk settings, features, and node-specific options. You can edit this file at boot time as described above.

Below is an annotated example configuration:

// This file is used to configure the device.
// The device will read this file and apply the settings during boot.
{
  "version": "1.5.5",
  "networking": {
    // Default CNI (for example, calico or cilium)
    "cni": {
      "type": "calico"
    },
    // Firewalld configuration
    "firewalld": {
      "enabled": "false"
    },
    "iface": {
      // Automatically select the network interface (found in eth cards)
      "dev": "auto",
      "ipconfig": "dhcp"
      // Uncomment and set the following for static IP configuration:
      // "ipconfig": "static",
      // "ipaddr": "192.168.0.100",
      // "gateway": "192.168.0.1",
      // "subnet_mask": "255.255.255.0",
      // "dns": "1.1.1.1,8.8.8.8",
      // "dns_search": ""
    }
  },

  // Disk configuration
  "disk": {
    // Root disk configuration:
    "root": {
      // Automatically choose the install disk
      "disk": "auto",
      // Size in MiB or percentage (minimum 12GiB, recommended 20% or higher)
      "size_mib": "20%"
    },
    // If set to true, LVM volumes will not be cleared before installation
    "keep_lvm" : "true"
  },

  // Features configuration
  "features": {
    "virt": {
      // Enable virtualization
      "enabled": "false",
      // Emulation options: "true", "false", or "auto"
      "emulation": "auto"
    },
    "argocd": {
      // Enable ArgoCD
      "enabled": "false"
    }
  },

  "node": {
    // Use hostname for resolving the control plane endpoint.
    "useHostname": "true",
    // Set a custom hostname for the control plane (if provided, useHostname is ignored)
    "customHostname": "",
    "ha": {
      // Network interface for the virtual IP (or "auto" to select the default route)
      "interface": "auto",
      // HA mode: "none", "keepalived", or "kubevip"
      "type": "none"
      // For external load balancer mode, you may configure:
      // "apiControlEndpoint": "192.168.0.200",
      // "apiControlEndpointSubnetSize": 24
    }
  }
}

Note: This file is editable during the first 10 seconds of boot by pressing 1 (nano) or 2 (vi).

Scripts Overview

All scripts are placed in the system PATH for easy access.

generate_join.sh

This script generates a join command that is used by new nodes to join the cluster.

  • Usage:

    generate_join.sh [worker|control] [--debug]

  • Parameters:

    • worker or control: Specifies whether the join command is for a worker node or a control plane node.
    • --debug: (Optional) Outputs the join command in clear text instead of base64 encoding it.

  • Functionality:

    • Generates a unique join token using kubeadm token create.
    • For control plane nodes, it uploads certificates and appends the certificate key to the join command.
    • By default, the join command is encoded in base64.
    • For worker nodes, the script also waits for the new node to be detected in the cluster, then labels it as a worker and deletes the join token.

Example:

# Generate a join command for a worker node (encoded)
generate_join.sh worker

# Generate a join command for a control plane node (clear text)
generate_join.sh control --debug

join_cluster.sh (Worker Version)

This script is executed on a new worker node to join the existing Kubernetes cluster.

  • Usage:

    join_cluster.sh '<base64_join_command>'

  • Process:

    • Decodes the provided join command from base64.
    • Executes the join command with root privileges.
    • Sets up kubeconfig for the root and core users.
    • Verifies the node has successfully joined the cluster by checking its hostname with kubectl get nodes.

Excerpt from the Worker Script:

#!/bin/bash
# Script to join a new node as a Worker to an existing Kubernetes cluster using an argument

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 '<join_command>'"
    exit 1
fi

source /usr/local/bin/k4all-utils
JOIN_COMMAND=$(echo "$1" | base64 -d)

sudo bash -c "$JOIN_COMMAND"

setup_kubeconfig_for_user "root" "/root" "kubelet.conf"
setup_kubeconfig_for_user "core" "/home/core" "kubelet.conf"

# Verification loop...

join_cluster.sh (Control Plane Version)

This variant is used on a new control plane node.

  • Usage:

    join_cluster.sh '<base64_join_command>'

  • Process:

    • Decodes and executes the join command.
    • Sets up kubeconfig for both root and core users.
    • Verifies that the node has joined the cluster.
    • Removes the default taint from the control plane node, allowing workloads to be scheduled if needed.

Excerpt from the Control Plane Script:

#!/bin/bash
# Script to join a new node as a control plane to an existing Kubernetes cluster using an argument

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 '<join_command>'"
    exit 1
fi

source /usr/local/bin/k4all-utils
JOIN_COMMAND=$(echo "$1" | base64 -d)

sudo bash -c "$JOIN_COMMAND"

setup_kubeconfig_for_user "root" "/root"
setup_kubeconfig_for_user "core" "/home/core"

# Verification and taint removal...

Summary

  • Bootstrap Node:
    The first node in the cluster, configured via an editable config.json and responsible for generating join tokens.

  • Join Process:
    Use generate_join.sh on the bootstrap node to generate a join command. Then, run join_cluster.sh on new nodes (with the appropriate version for worker or control plane) to add them to the cluster.

  • HA Modes:
    Configure your preferred HA mode (kubevip, keepalived, or external load balancer) in config.json during boot. Remember that kubevip should only be set on the bootstrap node.

  • Images:
    Ensure you download the correct images for bootstrap, worker, and control nodes from the repository releases.

By following these steps and using the provided scripts, you can easily deploy and scale your Kubernetes cluster with K4all.

Happy clustering!