This guide provides a streamlined approach to deploying and managing Kubernetes clusters on OpenNebula using CAPONE (Cluster API Provider for OpenNebula). CAPONE leverages the Kubernetes Cluster API (CAPI) to automate the provisioning of virtual machines (VMs), simplifying cluster lifecycle management.

With CAPONE, you can quickly spin up a Kubernetes cluster on OpenNebula, including:

• Virtual Network Functions (VNFs): A virtual router for external connectivity.
• Control Plane: Manages the cluster and runs the Kubernetes API.
• Worker Nodes: Run containerized workloads.

Before you begin, verify that your environment meets the following requirements:

• OpenNebula: Version ≥ 6.10 with OneGate enabled.
• Docker: Installed and running on your host.
• KinD: Installed for setting up the local management cluster.
• clusterctl: Installed for interacting with the Cluster API.
• kubectl: Installed to manage Kubernetes clusters.

Additionally, ensure that your VM resource allocations meet these recommendations (16GB RAM in the host - 8 vCPUs):

• VNF VM: 512 MB memory, 1 VCPU.
• Kubernetes Nodes: 3 GB memory, 1 VCPU each.

# Update package lists and install prerequisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Set up Docker’s official GPG key and keyring
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the Docker repository dynamically based on your Ubuntu version
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package lists and install the latest Docker version
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify installation
docker --version

# Get the latest version dynamically from GitHub
LATEST_KIND_VERSION=$(curl -s https://api.github.com/repos/kubernetes-sigs/kind/releases/latest | grep '"tag_name":' | cut -d '"' -f 4)

# Download and install KinD
curl -Lo ./kind "https://kind.sigs.k8s.io/dl/${LATEST_KIND_VERSION}/kind-linux-amd64"
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# Verify installation
kind version

# Get the latest clusterctl version dynamically from GitHub
LATEST_CLUSTERCTL_VERSION=$(curl -s https://api.github.com/repos/kubernetes-sigs/cluster-api/releases/latest | grep '"tag_name":' | cut -d '"' -f 4)

# Download and install clusterctl
curl -L "https://github.com/kubernetes-sigs/cluster-api/releases/download/${LATEST_CLUSTERCTL_VERSION}/clusterctl-linux-amd64" -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

# Verify installation
clusterctl version

sudo snap install kubectl --classic

# Verify the installation
kubectl version --client

CAPONE deployment requires two main pre-deployment configurations: setting up the required appliances and configuring the networking.

onemarketapp export "CAPONE VNF" capone131-vr -d 1
onemarketapp export "CAPONE K8s node" capone131 -d 1

# .env file
CCM_IMG=ghcr.io/opennebula/cloud-provider-opennebula:latest
CLUSTER_NAME=one
ONE_XMLRPC=http://212.47.235.99:2633/RPC2
ONE_AUTH=oneadmin:oneadmin
MACHINE_TEMPLATE_NAME=capone131
ROUTER_TEMPLATE_NAME=capone131-vr
PUBLIC_NETWORK_NAME=public
PRIVATE_NETWORK_NAME=private

# If left empty, OpenNebula will assign the IP automatically
CONTROL_PLANE_HOST=

CONTROL_PLANE_MACHINE_COUNT=1
WORKER_MACHINE_COUNT=2

Load and verify the environment variables:

set -a
source .env
set +a

env | grep -E 'CCM_IMG|CLUSTER_NAME|ONE_XMLRPC|ONE_AUTH|MACHINE_TEMPLATE_NAME|ROUTER_TEMPLATE_NAME|PUBLIC_NETWORK_NAME|PRIVATE_NETWORK_NAME'

1. Create a KinD Configuration File:

   Create a file called kind-config.yaml with the following content:

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
     - role: control-plane

2. Deploy the KinD Cluster:

   kind create cluster --name one --config kind-config.yaml
   kubectl cluster-info --context kind-one

• Create the clusterctl-config.yaml:

  providers:
    - name: opennebula
      type: InfrastructureProvider
      url: "https://github.com/OpenNebula/cluster-api-provider-opennebula/releases/download/latest/infrastructure-components.yaml"

• Initialize the management cluster using clusterctl with your configuration file:

  clusterctl init --config=clusterctl-config.yaml --infrastructure=opennebula:v0.1.0

clusterctl generate cluster ${CLUSTER_NAME} --infrastructure=opennebula --config=clusterctl-config.yaml | kubectl apply -f -
clusterctl get kubeconfig ${CLUSTER_NAME} > kubeconfig-${CLUSTER_NAME}.yaml

Flannel is required to enable pod-to-pod communication across nodes, as Kubernetes clusters lack a default networking solution.

kubectl --kubeconfig=kubeconfig-${CLUSTER_NAME}.yaml apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Check the status of your nodes:

kubectl --kubeconfig=kubeconfig-${CLUSTER_NAME}.yaml get nodes

You should see a list similar to:

NAME                   STATUS   ROLES           AGE     VERSION
one-g89nf              Ready    control-plane   5m      v1.31.4
one-md-0-cmq95-djz7d   Ready    <none>          4m27s   v1.31.4
one-md-0-cmq95-kjrrm   Ready    <none>          4m12s   v1.31.4

With oneadmin user, execute commands directly on the master (control plane) node:

ssh -A -J root@<Public-IP-VR> root@<Private-IP-CP>

Then, verify the nodes and pods:

kubectl get nodes

A CAPONE-deployed cluster typically consists of:

• Virtual Router (VNF): Provides external connectivity.
• Control Plane (CP): Runs Kubernetes management services.
• Worker Nodes: Execute your containerized applications.

An example mapping might look like:

CAPONE provides an easy way to manage Kubernetes clusters by clearly distinguishing between:

• Management Clusters: Local KinD clusters responsible for deploying and controlling workload clusters.
• Workload Clusters: Kubernetes clusters deployed on OpenNebula infrastructure.

The Management Cluster (KinD) controls the lifecycle of your Kubernetes workload clusters.

To see all KinD management clusters:

kind get clusters

To get more details about a specific cluster (e.g., one):

kubectl cluster-info --context kind-one

This will provide information about the cluster nodes, including their status, roles, and IPs.

To add another management cluster (e.g., named mgmt-2):

1. Create a KinD configuration file (management-cluster.yaml):

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
     - role: control-plane
   

2. Deploy the new KinD cluster:

   kind create cluster --name mgmt-2 --config management-cluster.yaml
   

To manage a different KinD management cluster (e.g., mgmt-2):

kubectl config use-context kind-mgmt-2

To remove an existing KinD management cluster (e.g., mgmt-2):

kind delete cluster --name mgmt-2

Workload Clusters are Kubernetes clusters deployed on OpenNebula. Here’s how to manage them:

To list all workload clusters managed by your current KinD Management Cluster:

kubectl get clusters --all-namespaces

To create a new workload cluster (e.g., new-workload):

1. Update the .env file with the new cluster name:

   CLUSTER_NAME=new-workload
   # Update other variables if necessary
   

2. Load environment variables:

   set -a
   source .env
   set +a
   

3. Deploy the workload cluster:

   clusterctl generate cluster ${CLUSTER_NAME} --infrastructure=opennebula --config=clusterctl-config.yaml | kubectl apply -f -
   

4. Retrieve the kubeconfig file:

   clusterctl get kubeconfig ${CLUSTER_NAME} > kubeconfig-${CLUSTER_NAME}.yaml
   

Flannel is required for pod-to-pod communication. Deploy it with:

kubectl --kubeconfig=kubeconfig-${CLUSTER_NAME}.yaml apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Ensure all nodes are up and running:

kubectl --kubeconfig=kubeconfig-${CLUSTER_NAME}.yaml get nodes

To scale up worker nodes (e.g., set to 3 replicas):

kubectl scale machinedeployment ${CLUSTER_NAME}-md-0 --replicas=3

Check the updated scaling status:

kubectl get machines -n ${CLUSTER_NAME}

To increase control plane nodes (e.g., set to 3 replicas):

kubectl scale kubeadmcontrolplane ${CLUSTER_NAME} --replicas=3

Check the updated scaling status:

kubectl get machines -n ${CLUSTER_NAME}

To completely remove a workload cluster (e.g., new-workload):

kubectl delete cluster ${CLUSTER_NAME}

Verify that the cluster is deleted:

kubectl get clusters --all-namespaces

CAPONE can deploy a virtual router as a Load Balancer (LB) to expose Kubernetes services externally.

In an LB-enabled setup, you typically have:

• Control Plane Virtual Router
• Load Balancer Virtual Router: With a dedicated floating IP.
• Kubernetes Control Plane and Worker Nodes: Running on the private network.

For example, your VMs may be mapped as:

Follow these steps to expose an Nginx deployment via a LoadBalancer service.

Create a file named nginx.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

Apply the application:

kubectl apply -f nginx.yaml

Check that the deployment is running:

kubectl get deployments

Then retrieve the external IP of the service:

kubectl get services

The nginx-service should now display an EXTERNAL-IP (e.g., 172.20.0.201). Test access to ensure your service is reachable.

By following these steps, you can deploy, scale, and manage your Kubernetes clusters on OpenNebula using CAPONE. This modular approach allows for flexibility, easy upgrades, and integration with load balancing for production workloads.