oneke_deploy - OpenNebula/one-apps GitHub Wiki

Deploying OneKE

In this section we'll focus on advanced topics when deploying your Kubernetes clusters:

  • High Availability (HA) deployments
  • Using affinity rules in HA deployments
  • Using Terraform to create your OneKE Kubernetes clusters
  • Customizing the Kubernetes components deployed with the cluster

For those seeking a quick start, we recommend referring to the Quick Start Guide, which provides step-by-step instructions to get up and running with OneKE. Additionally, for simple deployments using the Sunstone web UI, you can check Running Kubernetes Clusters in the OpenNebula documentation.

High Availability OneKE

Important

For high availability both Virtual IPs (VIPs) must be provided.

By default, the OneKE Virtual Appliance is preconfigured to work as a non-Highly-Available K8s cluster, since OneFlow Service Templates deploy each service role as a single VM. Kubernetes High-Availability involves setting up a Kubernetes cluster, along with its components, in such a way that there is no single point of failure. To achieve high-availability, the following OneKE components should be scaled up: VNF (at least 2 VMs), master (at least 3 VMs) and storage (at least 2 VMs). See figure below:

Public Network
─────────────────────────────────────────┬───────────────────────────────────────────────────────────
                                         β”‚ VIP0 / FLOATING_IP (3.27.6.5) (ep0.eth0.vr)
                       β”Œ ─ ─ ─ ─ ─ ─ ─ ─ β”Ό ─ ─ ┐
                       β”‚ OneKE vnf       β”‚     β”‚
                       β”‚ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”‚
                       β”‚ β”‚        β”‚ β”‚        β”‚ β”‚
                       β”‚ β”‚  VM-1  β”‚ β”‚  VM-2  β”‚ β”‚
                       β”‚ β”‚        β”‚ β”‚        β”‚ β”‚
                       β”‚ └──eth1β”€β”€β”˜ └──eth1β”€β”€β”˜ β”‚
                       β”‚     β”‚           β”‚     β”‚
                       β”‚  10.0.0.2    10.0.0.3 β”‚
                       β”‚                 β”‚     β”‚
                       β”” ─ ─ ─ ─ ─ ─ ─ ─ β”Ό ─ ─ β”˜
                                         β”‚ VIP1 / FLOATING_IP (10.0.0.1) (ep0.eth1.vr)
───────┬───────────┬─────────┬───────────┴──┬───────────┬─────────────┬───────────┬─────────┬────────
Private Network    β”‚         β”‚              β”‚           β”‚             β”‚           β”‚         β”‚
       β”‚           β”‚         β”‚              β”‚           β”‚             β”‚           β”‚         β”‚
 β”Œ ─ ─ β”Ό ─ ─ ─ ─ ─ β”Ό ─ ─ ─ ─ β”Ό ─ ─ ─┐ β”Œ ─ ─ β”Ό ─ ─ ─ ─ ─ β”Ό ─ ─ ┐ β”Œ ─ ─ β”Ό ─ ─ ─ ─ ─ β”Ό ─ ─ ─ ─ β”Ό ─ ─ ─┐
 β”‚ OneKE master    β”‚         β”‚      β”‚ β”‚ OneKE worker    β”‚     β”‚ β”‚ OneKE storage   β”‚         β”‚      β”‚
 β”‚     β”‚           β”‚         β”‚      β”‚ β”‚     β”‚           β”‚     β”‚ β”‚     β”‚           β”‚         β”‚      β”‚
 β”‚  10.0.0.4    10.0.0.5   10.0.0.6 β”‚ β”‚  10.0.0.7    10.0.0.8 β”‚ β”‚  10.0.0.9  10.0.0.10   10.0.0.11 β”‚
 β”‚     β”‚           β”‚         β”‚      β”‚ β”‚     β”‚           β”‚     β”‚ β”‚     β”‚           β”‚         β”‚      β”‚
 β”‚ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”‚ β”‚ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”‚ β”‚ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”Œβ”€β”€eth0──┐ β”‚
 β”‚ β”‚        β”‚ β”‚        β”‚ β”‚        β”‚ β”‚ β”‚ β”‚        β”‚ β”‚        β”‚ β”‚ β”‚ β”‚        β”‚ β”‚        β”‚ β”‚        β”‚ β”‚
 β”‚ β”‚  VM-3  β”‚ β”‚  VM-4  β”‚ β”‚  VM-5  β”‚ β”‚ β”‚ β”‚  VM-6  β”‚ β”‚  VM-7  β”‚ β”‚ β”‚ β”‚  VM-8  β”‚ β”‚  VM-9  β”‚ β”‚  VM-10 β”‚ β”‚
 β”‚ β”‚        β”‚ β”‚        β”‚ β”‚        β”‚ β”‚ β”‚ β”‚        β”‚ β”‚        β”‚ β”‚ β”‚ β”‚        β”‚ β”‚        β”‚ β”‚        β”‚ β”‚
 β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
 β”” ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ β”€β”˜ β”” ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ β”˜ β”” ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ β”€β”˜

Downloading and Deploying

  1. Download the Service OneKE 1.31 (or the airgapped version, Service OneKE 1.31 Airgapped) from the OpenNebula Marketplace:

    onemarketapp export 'Service OneKE 1.31' 'Service OneKE 1.31' --datastore 1
    

    Output should be similar to the following:

    $ onemarketapp export 'Service OneKE 1.31' 'Service OneKE 1.31' --datastore 1
    IMAGE
        ID: 202
        ID: 203
        ID: 204
    VMTEMPLATE
        ID: 204
        ID: 205
        ID: 206
    SERVICE_TEMPLATE
        ID: 104
  2. Adjust Flow and VM templates to meet you needs. (You may want to increase CPU and MEMORY for workers, add HOT_RESIZE=[...] or setup anti-affined VM groups, etc.)

    $ oneflow-template update 104
    $ onetemplate update 205
  3. Instantiate the Flow service:

    $ cat >/tmp/OneKE-instantiate <<'EOF'
    {
        "name": "OneKE/1",
        "networks_values": [
            {"Public": {"id": "0"}},
            {"Private": {"id": "1"}}
        ],
        "custom_attrs_values": {
            "ONEAPP_VROUTER_ETH0_VIP0": "3.27.6.5",
            "ONEAPP_VROUTER_ETH1_VIP0": "10.0.0.1",
    
            "ONEAPP_RKE2_SUPERVISOR_EP": "",
            "ONEAPP_K8S_CONTROL_PLANE_EP": "",
            "ONEAPP_K8S_EXTRA_SANS": "localhost,127.0.0.1,k8s.yourdomain.it",
    
            "ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED": "YES",
    
            "ONEAPP_K8S_HTTP_PROXY": "",
            "ONEAPP_K8S_HTTPS_PROXY": "",
            "ONEAPP_K8S_NO_PROXY": "",
    
            "ONEAPP_K8S_MULTUS_ENABLED": "NO",
            "ONEAPP_K8S_MULTUS_CONFIG": "",
            "ONEAPP_K8S_CNI_PLUGIN": "cilium",
            "ONEAPP_K8S_CNI_CONFIG": "",
            "ONEAPP_K8S_CILIUM_BGP_ENABLED": "YES",
            "ONEAPP_K8S_CILIUM_RANGE": "",
    
            "ONEAPP_K8S_METALLB_ENABLED": "NO",
            "ONEAPP_K8S_METALLB_CLASS" : "",
            "ONEAPP_K8S_METALLB_CONFIG": "",
            "ONEAPP_K8S_METALLB_RANGE": "",
    
            "ONEAPP_K8S_LONGHORN_ENABLED": "YES",
            "ONEAPP_STORAGE_DEVICE": "/dev/vdb",
            "ONEAPP_STORAGE_FILESYSTEM": "xfs",
    
            "ONEAPP_K8S_TRAEFIK_ENABLED": "YES",
            "ONEAPP_VNF_HAPROXY_INTERFACES": "eth0",
            "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "30",
            "ONEAPP_VNF_HAPROXY_LB0_PORT": "9345",
            "ONEAPP_VNF_HAPROXY_LB1_PORT": "6443",
            "ONEAPP_VNF_HAPROXY_LB2_PORT": "443",
            "ONEAPP_VNF_HAPROXY_LB3_PORT": "80",
    
            "ONEAPP_VNF_DNS_ENABLED": "YES",
            "ONEAPP_VNF_DNS_INTERFACES": "eth1",
            "ONEAPP_VNF_DNS_NAMESERVERS": "1.1.1.1,8.8.8.8",
            "ONEAPP_VNF_NAT4_ENABLED": "YES",
            "ONEAPP_VNF_NAT4_INTERFACES_OUT": "eth0",
            "ONEAPP_VNF_ROUTER4_ENABLED": "YES",
            "ONEAPP_VNF_ROUTER4_INTERFACES": "eth0,eth1"
        }
    }
    EOF
    $ oneflow-template instantiate 'Service OneKE 1.31' /tmp/OneKE-instantiate
    ID: 105

    After successful deployment the Flow service should display RUNNING state:

    $ oneflow list
      ID USER     GROUP    NAME                           STARTTIME STAT
     105 oneadmin oneadmin Service OneKE 1.31        02/15 09:46:56 RUNNING
  4. Scale Flow roles (if required):

    $ oneflow scale 105 vnf 2
    $ oneflow scale 105 master 3
    $ oneflow scale 105 worker 2
    $ oneflow scale 105 storage 3

Note

By default, the cardinality of the storage role is set to 0 to allow deployments with Longhorn disabled.

Note

You can set the cardinality of each role in step 2 when you update the template.

Warning

You can scale the master role up to an odd number of masters, but be careful if scaling down as it may break your cluster. If you require multi-master HA, start with a single master, then scale up to 3 and keep it that way.

After a while we can examine the service log:

$ oneflow show 'OneKE/1'
...
LOG MESSAGES
05/13/24 12:30 [I] New state: DEPLOYING_NETS
05/13/24 12:30 [I] New state: DEPLOYING
05/13/24 12:39 [I] New state: RUNNING
05/13/24 12:43 [I] Role master scaling up from 1 to 3 nodes
05/13/24 12:43 [I] New state: SCALING
05/13/24 12:52 [I] New state: COOLDOWN
05/13/24 12:55 [I] New state: RUNNING

And we can list the cluster nodes using kubectl:

$ kubectl get nodes
NAME                    STATUS   ROLES                       AGE     VERSION
oneke-ip-172-20-0-101   Ready    control-plane,etcd,master   31m     v1.31.3+rke2r1
oneke-ip-172-20-0-102   Ready    <none>                      28m     v1.31.3+rke2r1
oneke-ip-172-20-0-103   Ready    <none>                      28m     v1.31.3+rke2r1
oneke-ip-172-20-0-104   Ready    control-plane,etcd,master   11m     v1.31.3+rke2r1
oneke-ip-172-20-0-105   Ready    control-plane,etcd,master   10m     v1.31.3+rke2r1

Warning

Please plan ahead and avoid scaling down master and storage roles as it may break ETCD's quorum or cause data loss. There is no obvious restriction for the worker role, however; it can be safely rescaled at will.

Deploying the cluster behind an egress traffic proxy

In case we want to deploy the OneKE cluster behind an egress traffic HTTP/S proxy (e.g. connections made when downloading container images or bootstrapping the cluster nodes), we can set the following service template custom variables:

  • ONEAPP_K8S_HTTP_PROXY: Proxy address to use when initiating cluster HTTP connection(s).
  • ONEAPP_K8S_HTTP_PROXY: Proxy address to use when initiating cluster HTTPS connection(s).
  • ONEAPP_K8S_NO_PROXY: Network address(es), network address range(s) and domains to exclude from using the proxy when initiating cluster connection(s). If not specified, the following addresses are set: localhost,127.0.0.1/32,$ONEAPP_RKE2_SUPERVISOR_EP,$ONEAPP_K8S_CONTROL_PLANE_EP

More information in the rke2 documentation.

Anti-affinity

In an HA configuration, VMs related to the same role should be scheduled on different physical hosts to guarantee HA in case of a host failure. OpenNebula provides VM Group resources to achieve proper Host/VM affinity/anti-affinity.

In the following section, we provide an example of how to create VM Group resources and how to modify OneKE's OneFlow Service Template to include VM groups.

Let's assume that epsilon and omicron are hosts we want to use to deploy OneKE; a VM Group may be created as shown below:

$ cat >/tmp/OneKE-vmgroup <<'EOF'
NAME = "Service OneKE 1.31"
ROLE = [
    NAME         = "vnf",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "master",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "worker",
    HOST_AFFINED = "epsilon,omicron"
]
ROLE = [
    NAME         = "storage",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
EOF

$ onevmgroup create /tmp/OneKE-vmgroup
ID: 1

Important

For the worker role, POLICY is not defined, which allows you to reuse hosts multiple times.

Now we can modify the OneKE OneFlow Service Template. In the example below we'll pipe the template in JSON format and modify it with the jq command-line JSON processor.

$ oneflow-template show 'Service OneKE 1.31' --json | >/tmp/OneKE-update.json jq -r --arg vmgroup 'Service OneKE 1.31' -f /dev/fd/3 3<<'EOF'
.DOCUMENT.TEMPLATE.BODY | del(.registration_time) | . += {
  roles: .roles | map(
    .vm_template_contents = "VMGROUP=[VMGROUP_NAME=\"\($vmgroup)\",ROLE=\"\(.name)\"]\n" + .vm_template_contents
  )
}
EOF

The above command adds the template contents for the VMs in the OneKE service. The contents of the updated file, /tmp/OneKE-update.json, are shown below:

{
  "name": "Service OneKE 1.31",
  "deployment": "straight",
  "description": "",
  "roles": [
    {
      "name": "vnf",
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.31\",ROLE=\"vnf\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Public\" ]\nNIC = [\n  NAME = \"NIC1\",\n  NETWORK_ID = \"$Private\" ]\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_VNF_HAPROXY_INTERFACES = \"$ONEAPP_VNF_HAPROXY_INTERFACES\"\nONEAPP_VNF_HAPROXY_REFRESH_RATE = \"$ONEAPP_VNF_HAPROXY_REFRESH_RATE\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\nONEAPP_VNF_DNS_INTERFACES = \"$ONEAPP_VNF_DNS_INTERFACES\"\nONEAPP_VNF_DNS_NAMESERVERS = \"$ONEAPP_VNF_DNS_NAMESERVERS\"\nONEAPP_VNF_NAT4_ENABLED = \"$ONEAPP_VNF_NAT4_ENABLED\"\nONEAPP_VNF_NAT4_INTERFACES_OUT = \"$ONEAPP_VNF_NAT4_INTERFACES_OUT\"\nONEAPP_VNF_ROUTER4_ENABLED = \"$ONEAPP_VNF_ROUTER4_ENABLED\"\nONEAPP_VNF_ROUTER4_INTERFACES = \"$ONEAPP_VNF_ROUTER4_INTERFACES\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 0
    },
    {
      "name": "master",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.31\",ROLE=\"master\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_EXTRA_SANS = \"$ONEAPP_K8S_EXTRA_SANS\"\nONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED = \"$ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED\"\nONEAPP_K8S_HTTP_PROXY = \"$ONEAPP_K8S_HTTP_PROXY\"\nONEAPP_K8S_HTTPS_PROXY = \"$ONEAPP_K8S_HTTPS_PROXY\"\nONEAPP_K8S_NO_PROXY = \"$ONEAPP_K8S_NO_PROXY\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_MULTUS_CONFIG = \"$ONEAPP_K8S_MULTUS_CONFIG\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_CNI_CONFIG = \"$ONEAPP_K8S_CNI_CONFIG\"\nONEAPP_K8S_CILIUM_BGP_ENABLED = \"$ONEAPP_K8S_CILIUM_BGP_ENABLED\"\nONEAPP_K8S_CILIUM_RANGE = \"$ONEAPP_K8S_CILIUM_RANGE\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_METALLB_CLASS = \"$ONEAPP_K8S_METALLB_CLASS\"\nONEAPP_K8S_METALLB_CONFIG = \"$ONEAPP_K8S_METALLB_CONFIG\"\nONEAPP_K8S_METALLB_RANGE = \"$ONEAPP_K8S_METALLB_RANGE\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 1
    },
    {
      "name": "worker",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.31\",ROLE=\"worker\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED = \"$ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED\"\nONEAPP_K8S_HTTP_PROXY = \"$ONEAPP_K8S_HTTP_PROXY\"\nONEAPP_K8S_HTTPS_PROXY = \"$ONEAPP_K8S_HTTPS_PROXY\"\nONEAPP_K8S_NO_PROXY = \"$ONEAPP_K8S_NO_PROXY\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 1
    },
    {
      "name": "storage",
      "parents": [
        "vnf"
      ],
      "cardinality": 0,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.31\",ROLE=\"storage\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_HTTP_PROXY = \"$ONEAPP_K8S_HTTP_PROXY\"\nONEAPP_K8S_HTTPS_PROXY = \"$ONEAPP_K8S_HTTPS_PROXY\"\nONEAPP_K8S_NO_PROXY = \"$ONEAPP_K8S_NO_PROXY\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_STORAGE_DEVICE = \"$ONEAPP_STORAGE_DEVICE\"\nONEAPP_STORAGE_FILESYSTEM = \"$ONEAPP_STORAGE_FILESYSTEM\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 2
    }
  ],
  "networks": {
    "Public": "M|network|Public||id:",
    "Private": "M|network|Private||id:"
  },
  "custom_attrs": {
    "ONEAPP_VROUTER_ETH0_VIP0": "O|text|Control Plane Endpoint VIP (IPv4)||",
    "ONEAPP_VROUTER_ETH1_VIP0": "O|text|Default Gateway VIP (IPv4)||",
    "ONEAPP_RKE2_SUPERVISOR_EP": "O|text|RKE2 Supervisor endpoint||ep0.eth0.vr:9345",
    "ONEAPP_K8S_CONTROL_PLANE_EP": "O|text|Control Plane endpoint||ep0.eth0.vr:6443",
    "ONEAPP_K8S_EXTRA_SANS": "O|text|ApiServer extra certificate SANs||localhost,127.0.0.1,ep0.eth0.vr,${vnf.TEMPLATE.CONTEXT.ETH0_IP}",
    "ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED": "O|boolean|Enables default RKE2 cloud controller||YES",
    "ONEAPP_K8S_HTTP_PROXY": "O|text|Defines HTTP Proxy for cluster traffic (default none)||",
    "ONEAPP_K8S_HTTPS_PROXY": "O|text|Defines HTTPS Proxy for cluster traffic (default none)||",
    "ONEAPP_K8S_NO_PROXY": "O|text|Defines non-proxied exceptions for cluster traffic (only effective if jhttp proxy is enabled)||localhost,127.0.0.1/32,$ONEAPP_RKE2_SUPERVISOR_EP,$ONEAPP_K8S_CONTROL_PLANE_EP",
    "ONEAPP_K8S_MULTUS_ENABLED": "O|boolean|Enable Multus||NO",
    "ONEAPP_K8S_MULTUS_CONFIG": "O|text64|Multus custom config (default none)||",
    "ONEAPP_K8S_CNI_PLUGIN": "O|list|CNI plugin supported by RKE2|canal,calico,cilium|cilium",
    "ONEAPP_K8S_CNI_CONFIG": "O|text64|CNI custom config (default none)||",
    "ONEAPP_K8S_CILIUM_BGP_ENABLED": "O|boolean|Enable BGP control plane for Cilium. It could conflict with other Service LB Controllers.||",
    "ONEAPP_K8S_CILIUM_RANGE": "O|text|Cilium LB IP CIDR. Only effective if `ONEAPP_K8S_CILIUM_BGP_ENABLED` is set to `YES` or undefined. (default none)||",
    "ONEAPP_K8S_METALLB_ENABLED": "O|boolean|Enable MetalLB||NO",
    "ONEAPP_K8S_METALLB_CLASS": "O|text|MetalLB Load Balancer Class name||metallb",
    "ONEAPP_K8S_METALLB_RANGE": "O|text|MetalLB IP range (default none)||",
    "ONEAPP_K8S_METALLB_CONFIG": "O|text64|MetalLB custom config (default none)||",
    "ONEAPP_K8S_LONGHORN_ENABLED": "O|boolean|Enable Longhorn||NO",
    "ONEAPP_STORAGE_DEVICE": "O|text|Storage device path||/dev/vdb",
    "ONEAPP_STORAGE_FILESYSTEM": "O|text|Storage device filesystem||xfs",
    "ONEAPP_K8S_TRAEFIK_ENABLED": "O|boolean|Enable Traefik||NO",
    "ONEAPP_VNF_HAPROXY_INTERFACES": "O|text|Interfaces to run Haproxy on||eth0",
    "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "O|number|Haproxy refresh rate||30",
    "ONEAPP_VNF_HAPROXY_LB0_PORT": "O|number|HTTPS RKE2 Supervisor port||9345",
    "ONEAPP_VNF_HAPROXY_LB1_PORT": "O|number|HTTPS API Server port||6443",
    "ONEAPP_VNF_HAPROXY_LB2_PORT": "O|number|HTTPS ingress port||443",
    "ONEAPP_VNF_HAPROXY_LB3_PORT": "O|number|HTTP ingress port||80",
    "ONEAPP_VNF_DNS_ENABLED": "O|boolean|Enable DNS recursor||YES",
    "ONEAPP_VNF_DNS_INTERFACES": "O|text|DNS - Interfaces||eth1",
    "ONEAPP_VNF_DNS_NAMESERVERS": "O|text|DNS - Nameservers||1.1.1.1,8.8.8.8",
    "ONEAPP_VNF_NAT4_ENABLED": "O|boolean|Enable NAT||YES",
    "ONEAPP_VNF_NAT4_INTERFACES_OUT": "O|text|NAT - Outgoing Interfaces||eth0",
    "ONEAPP_VNF_ROUTER4_ENABLED": "O|boolean|Enable Router||YES",
    "ONEAPP_VNF_ROUTER4_INTERFACES": "O|text|Router - Interfaces||eth0,eth1"
  },
  "ready_status_gate": true
}

Note

In the jq command we removed the registration_time key from the document as it is immutable.

Next, let's update the template:

$ oneflow-template update 'Service OneKE 1.31' /tmp/OneKE-update.json

Deploying with Terraform

Note

For information about the OpenNebula Terraform Provider please see the documentation.

Below is an example HCL2 single-file config to deploy a basic non-Airgapped, non-HA OneKE instance:

terraform {
  required_providers {
    opennebula = {
      source = "OpenNebula/opennebula"
      version = ">= 1.4.0"
    }
  }
}

variable "one_user" {
  type = string
  description = "User account name to authenticate with OpenNebula"
#  default = "oneadmin"
}

variable "one_password" {
  type = string
  description = "Password for the user account to authenticate with OpenNebula"
}

variable "one_host" {
  type = string
  description = "Hostname or IP of the host where OpenNebula is running"
#  default = "localhost"
}

data "http" "appliances" {
  url = "https://marketplace.opennebula.io/appliance"
  request_headers = {
    User-Agent = "OpenNebula 6.8.0"
    Accept     = "application/json"
  }
}

locals {
  appliances = {
    for a in jsondecode(data.http.appliances.response_body).appliances : a.name => a
  }

  name = "Service OneKE 1.31"

  service = local.appliances[local.name]

  roles = {
    for k, v in local.service.roles : k => merge(local.appliances[v], {
      opennebula_template = jsondecode(local.appliances[v].opennebula_template)
    })
  }

  md5_to_url = {
    for f in distinct(flatten([
      for r in values(local.roles) : concat(
        try(r.files, []),
        [for d in try(r.disks, []) : local.appliances[d].files]
      )
    ])) : f.md5 => f.url
  }

  role_to_md5 = {
    for k, v in local.roles : k => [
      for f in flatten(concat(
        try(v.files, []),
        [for d in try(v.disks, []) : local.appliances[d].files]
      )) : f.md5
    ]
  }
}

provider "opennebula" {
  endpoint      = "http://${var.one_host}:2633/RPC2"
  flow_endpoint = "http://${var.one_host}:2474"
  username      = var.one_user
  password      = var.one_password
}

resource "random_id" "oneke" {
  byte_length = 4
}

resource "opennebula_image" "oneke" {
  for_each     = local.md5_to_url
  name         = "${local.name} ${each.key} ${random_id.oneke.id}"
  datastore_id = 1
  persistent   = false
  permissions  = 642
  dev_prefix   = "vd"
  driver       = "qcow2"
  path         = each.value
}

resource "opennebula_template" "oneke" {
  for_each = local.roles
  name     = "${local.name} ${each.key} ${random_id.oneke.id}"
  cpu      = try(each.value["opennebula_template"].CPU, null)
  vcpu     = try(each.value["opennebula_template"].VCPU, null)
  memory   = try(each.value["opennebula_template"].MEMORY, null)
  context  = try(each.value["opennebula_template"].CONTEXT, null)

  dynamic "graphics" {
    for_each = try([each.value["opennebula_template"].GRAPHICS], [])
    content {
      type   = try(graphics.value.TYPE, null)
      listen = try(graphics.value.LISTEN, null)
    }
  }

  dynamic "os" {
    for_each = try([each.value["opennebula_template"].OS], [])
    content {
      arch = try(os.value.ARCH, null)
      boot = "disk0"
    }
  }

  dynamic "disk" {
    for_each = local.role_to_md5[each.key]
    content {
      image_id = opennebula_image.oneke[disk.value].id
    }
  }
}

resource "opennebula_service_template" "oneke" {
  name        = "${local.name} ${random_id.oneke.id}"
  permissions = 642
  uname       = "oneadmin"
  gname       = "oneadmin"
  template = jsonencode({ "TEMPLATE" = { "BODY" = merge(
    jsondecode(local.service["opennebula_template"]),
    {
      "roles" : [
        for r in jsondecode(local.service["opennebula_template"]).roles : merge(
          r,
          { vm_template = tonumber(opennebula_template.oneke[r.name].id) }
        )
      ]
    }
  ) } })
}

resource "opennebula_service" "oneke" {
  name        = "${local.name} ${random_id.oneke.id}"
  template_id = opennebula_service_template.oneke.id
  extra_template = jsonencode({
    networks_values = [
      { Public = { id = "0" } },
      { Private = { id = "1" } },
    ]
    custom_attrs_values = {
      ONEAPP_VROUTER_ETH0_VIP0 = ""
      ONEAPP_VROUTER_ETH1_VIP0 = ""

      ONEAPP_RKE2_SUPERVISOR_EP   = "ep0.eth0.vr:9345"
      ONEAPP_K8S_CONTROL_PLANE_EP = "ep0.eth0.vr:6443"
      ONEAPP_K8S_EXTRA_SANS       = "localhost,127.0.0.1,ep0.eth0.vr,$${vnf.TEMPLATE.CONTEXT.ETH0_IP}"

      ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED = "YES"

      ONEAPP_K8S_HTTP_PROXY  = ""
      ONEAPP_K8S_HTTPS_PROXY = ""
      ONEAPP_K8S_NO_PROXY    = ""

      ONEAPP_K8S_MULTUS_ENABLED = "NO"
      ONEAPP_K8S_MULTUS_CONFIG  = ""

      ONEAPP_K8S_CNI_PLUGIN        = "cilium"
      ONEAPP_K8S_CNI_CONFIG        = ""
      ONEAPP_K8S_CILIUM_BGP_ENABLED = "YES"
      ONEAPP_K8S_CILIUM_RANGE      = ""

      ONEAPP_K8S_METALLB_ENABLED = "NO"
      ONEAPP_K8S_METALLB_CLASS   = ""
      ONEAPP_K8S_METALLB_CONFIG  = ""
      ONEAPP_K8S_METALLB_RANGE   = ""

      ONEAPP_K8S_LONGHORN_ENABLED = "YES"
      ONEAPP_STORAGE_DEVICE       = "/dev/vdb"
      ONEAPP_STORAGE_FILESYSTEM   = "xfs"

      ONEAPP_K8S_TRAEFIK_ENABLED      = "YES"
      ONEAPP_VNF_HAPROXY_INTERFACES   = "eth0"
      ONEAPP_VNF_HAPROXY_REFRESH_RATE = "30"
      ONEAPP_VNF_HAPROXY_LB0_PORT     = "9345"
      ONEAPP_VNF_HAPROXY_LB1_PORT     = "6443"
      ONEAPP_VNF_HAPROXY_LB2_PORT     = "443"
      ONEAPP_VNF_HAPROXY_LB3_PORT     = "80"

      ONEAPP_VNF_DNS_ENABLED         = "YES"
      ONEAPP_VNF_DNS_INTERFACES      = "eth1"
      ONEAPP_VNF_DNS_NAMESERVERS     = "1.1.1.1,8.8.8.8"
      ONEAPP_VNF_NAT4_ENABLED        = "YES"
      ONEAPP_VNF_NAT4_INTERFACES_OUT = "eth0"
      ONEAPP_VNF_ROUTER4_ENABLED     = "YES"
      ONEAPP_VNF_ROUTER4_INTERFACES  = "eth0,eth1"
    }
  })
  timeouts {
    create = "10m"
    delete = "5m"
  }
}

Important

The above script takes care of downloading templates and images; no preliminary oneflow export operation is required.

Customizing the Deployment

It is possible to modify VM templates related to the OneKE Virtual Appliance in order to customize the deployment, for example by adding more VM memory or VCPU cores to the workers, and resizing the disks for the storage nodes. This should be done before the creation of the K8s cluster, i.e. before instantiating the OneKE OneFlow Service Template.

When instantiating OneKE's OneFlow Service Template, you can further customize the deployment using the following custom attributes:

Parameter Default Roles Description Example
ONEAPP_VROUTER_ETH0_VIP0 all Control Plane Endpoint VIP (IPv4) 192.168.1.100
ONEAPP_VROUTER_ETH1_VIP0 all Default Gateway VIP (IPv4) 192.168.1.1
ONEAPP_RKE2_SUPERVISOR_EP ep0.eth0.vr:9345 master RKE2 Supervisor endpoint 192.168.1.101:9345
ONEAPP_K8S_CONTROL_PLANE_EP ep0.eth0.vr:6443 !vnf Control Plane endpoint 192.168.1.105:6443
ONEAPP_K8S_EXTRA_SANS localhost,
127.0.0.1,
ep0.eth0.vr,
${vnf.TEMPLATE.CONTEXT.ETH0_IP}
master ApiServer extra certificate SANs localhost,127.0.0.1,cluster.local
ONEAPP_RKE2_CLOUD_CONTROLLER_ENABLED YES !vnf Enables default RKE2 cloud controller (YES,NO)
ONEAPP_K8S_HTTP_PROXY !vnf Defines HTTP Proxy for cluster traffic http://your-proxy.example.com:8888
ONEAPP_K8S_HTTPS_PROXY !vnf Defines HTTPS Proxy for cluster traffic https://your-proxy.example.com:4443
ONEAPP_K8S_NO_PROXY localhost,
127.0.0.1/32,
$ONEAPP_RKE2_SUPERVISOR_EP,
$ONEAPP_K8S_CONTROL_PLANE_EP
!vnf Defines non-proxied exceptions for cluster traffic (only effective if HTTP proxy is enabled) localhost,127.0.0.0/8,10.0.0.0/8,cluster.local
ONEAPP_K8S_MULTUS_ENABLED NO all Enable Multus (YES,NO)
ONEAPP_K8S_MULTUS_CONFIG master Multus custom config (YAML Manifest encoded in base64) manifest (should be encoded in base64)
ONEAPP_K8S_CNI_PLUGIN cilium all CNI plugin (canal, calico or cilium)
ONEAPP_K8S_CNI_CONFIG master CNI custom config (YAML Manifest encoded in base64) canal, cilium, calico (should be encoded in base64)
ONEAPP_K8S_CILIUM_BGP_ENABLED NO master Enable BGP control plane for Cilium (YES,NO). It could conflict with other Service LB Controllers
ONEAPP_K8S_CILIUM_RANGE master Cilium LB IP/CIDR. Only effective if ONEAPP_K8S_CILIUM_BGP_ENABLED is set to YES or undefined 192.168.200.0/24
ONEAPP_K8S_METALLB_ENABLED NO master,worker Enable MetalLB (YES,NO)
ONEAPP_K8S_METALLB_CLASS metallb master MetalLB Load Balancer Class name my-metallb-service
ONEAPP_K8S_METALLB_RANGE master MetalLB IP range 192.168.150.50-192.168.150.80
ONEAPP_K8S_METALLB_CONFIG master MetalLB custom config (YAML Manifest encoded in base64) manifest (should be encoded in base64)
ONEAPP_K8S_LONGHORN_ENABLED NO all Enable Longhorn (YES,NO)
ONEAPP_STORAGE_DEVICE /dev/vdb storage Dedicated storage device for Longhorn
ONEAPP_STORAGE_FILESYSTEM xfs storage Filesystem type to init dedicated storage device
ONEAPP_K8S_TRAEFIK_ENABLED NO master,worker Enable Traefik (YES,NO)
ONEAPP_VNF_HAPROXY_INTERFACES eth0 vnf Interfaces to run HAProxy on
ONEAPP_VNF_HAPROXY_REFRESH_RATE 30 vnf HAProxy / OneGate refresh rate (in seconds)
ONEAPP_VNF_HAPROXY_LB0_PORT 9345 vnf,master HTTPS RKE2 Supervisor port
ONEAPP_VNF_HAPROXY_LB1_PORT 6443 vnf,master HTTPS API Server port
ONEAPP_VNF_HAPROXY_LB2_PORT 443 vnf,worker HTTPS ingress port
ONEAPP_VNF_HAPROXY_LB3_PORT 80 vnf,worker HTTP ingress port
ONEAPP_VNF_DNS_ENABLED YES all Enable DNS recursor (YES,NO)
ONEAPP_VNF_DNS_INTERFACES eth1 vnf DNS - Interfaces 'eth0 eth1 eth1 eth2 eth3/40.50.60.70'
ONEAPP_VNF_DNS_NAMESERVERS 1.1.1.1,8.8.8.8 vnf DNS - Nameservers
ONEAPP_VNF_NAT4_ENABLED YES vnf Enable NAT for the whole cluster (YES,NO)
ONEAPP_VNF_NAT4_INTERFACES_OUT eth0 vnf NAT - Outgoing (public) interfaces 'eth0 eth1'
ONEAPP_VNF_ROUTER4_ENABLED YES vnf Enable IPv4 forwarding for selected NICs (YES,NO)
ONEAPP_VNF_ROUTER4_INTERFACES eth0,eth1 vnf IPv4 Router - NICs selected for IPv4 forwarding eth0,eth1,eth2

Important

Virtual IP Parameters

ONEAPP_VROUTER_ETH0_VIP0: The VNF cluster uses this VIP to bind and expose the Kubernetes API port 6443 and RKE2's management port 9345. The eth0 NIC should be connected (routed or NATed) to the public subnet. Note that you can leave this parameter empty; in this case OneKE will assume this is a non-HA configuration, and will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

ONEAPP_VROUTER_ETH0_VIP0: The VNF cluster uses this VIP to act for every VM deployed in the private subnet. The eth1 NIC should be connected to the private subnet. Note that you can leave this parameter empty; in this case OneKE will assume this is a non-HA configuration, and will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

If you enable Longhorn (via ONEAPP_K8S_LONGHORN_ENABLED), you will also need to scale up the storage role or modify the cardinality parameter in the service template.

⚠️ **GitHub.com Fallback** ⚠️