Cluster setup w kuma mesh and kong gateway - JensvandeWiel/k3s-cluster-setup GitHub Wiki

This guide handles installing k3s, helm, cert-manager, external-dns, Kong ingress/gateway and kuma mesh.

Install k3s (Without Default Traefik Proxy)

  • To install k3s and disable the default Traefik proxy in one step:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -
  • The --disable=traefik flag ensures that the default Traefik proxy is not installed, allowing you to install and configure a custom version of Traefik.
  • After installation, export the KUBECONFIG environment variable to manage the cluster:
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Install Helm

  • Helm simplifies deploying applications on Kubernetes via charts.

Add the Helm repository and install Helm:

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

Install Kuma Service Mesh

  • Kuma will handle east-west (service-to-service) traffic with strict mTLS. We will also use kuma CNI so that the sidecars don't need a InitContainer relying on root privileges.

Install Kuma using Helm:

helm repo add kuma https://kumahq.github.io/charts
helm repo update
helm install kuma kuma/kuma \
  --namespace kuma-system \
  --create-namespace \
  --set controlPlane.mode=standalone \
  --set dataPlane.kdsGlobal.enabled=false \
  --set cni.enabled=true \
  --set cni.chained=true \
  --set cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d \
  --set cni.binDir=/var/lib/rancher/k3s/data/cni \
  --set cni.confName=10-flannel.conflist

Enable mTLS in the mesh:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
      mode: STRICT

(Optional) Allow all traffic in the mesh

  • You could also skip this and add more selective permissions when deploying the example app.
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: allow-all
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  from:
  - targetRef:
      kind: Mesh
    default:
      action: Allow

Install Kong Gateway/Ingress

Install the Gateway API CRDs before installing Kong Ingress Controller.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml

Create  a GatewayClass instance to use.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: kong
 annotations:
   konghq.com/gatewayclass-unmanaged: 'true'

spec:
 controllerName: konghq.com/kic-gateway-controller

Add Kong helm charts

helm repo add kong https://charts.konghq.com
helm repo update

Install Kong Ingress Controller and Kong Gateway with Helm:

helm install kong kong/ingress -n kong --create-namespace 

Enable sidecar injection on the kong namespace

kubectl label namespace kong kuma.io/sidecar-injection=enabled

Restart both the controller and the gateway to leverage sidecar injection

kubectl rollout restart -n kong deployment kong-gateway kong-controller

Install longhorn

Add the Longhorn helm repository:

helm repo add longhorn https://charts.longhorn.io
helm repo update

Install longhorn

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.8.0

Install ExternalDNS with Cloudflare

  • ExternalDNS automates the management of DNS records based on your Kubernetes resources.

Add the Bitnami Helm repository:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Install ExternalDNS with the Cloudflare provider:

helm install external-dns bitnami/external-dns \
--create-namespace \
--namespace external-dns \
--set provider=cloudflare \
--set cloudflare.apiToken=<YOUR_CLOUDFLARE_API_TOKEN> \
--set policy=sync \
--set txtOwnerId=external-dns-k3s \
--set cloudflare.proxied=false \
--set dnsRecordsPerPage=500 \
--set sources[0]=service \
--set sources[1]=ingress \
--set sources[2]=gateway-httproute \
--set sources[3]=gateway-grpcroute \
--set sources[4]=gateway-tcproute \
--set sources[5]=gateway-tlsroute \
--set sources[6]=gateway-udproute 
  • Replace <YOUR_CLOUDFLARE_API_TOKEN> with your Cloudflare API token.

Install Cert-Manager

  • Cert-Manager automates the creation and renewal of TLS/SSL certificates.

Add the Jetstack Helm repository and install Cert-Manager:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --set "extraArgs={--enable-gateway-api}"

Create (Cluster)Issuers for Let’s Encrypt:

  • You can choose between issuers, we will show how to create a staging cluster issuer, production issuer and a normal DNS issuer in the default namespace. You can choose between any of these, or use multiple.
Staging Issuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
	# Replace with your value
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: kong
Production Issuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
	# Replace with your value
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: kong
Create Cloudflare DNS issuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: dns-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
	# Replace with your value
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: dns-issuer
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

Deploy a Test Application using Gateway API

  • Deploy a simple "Hello World" application and configure it with Gateway API and Kong

Deploy the manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
    kuma.io/sidecar-injection: enabled
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
        kuma.io/sidecar-injection: enabled
    spec:
      containers:
      - name: hello-world
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: hello-world-gateway
spec:
  gatewayClassName: kong
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: hello-world-route
spec:
  parentRefs:
  - name: hello-world-gateway
  hostnames:
  # Replace with your value
  - "test.alpacaislands.nl"
  rules:
  - backendRefs:
    - name: hello-world
      port: 80

Allow traffic from gateway to the service

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  namespace: default
  name: hello-world-permission
spec:
  targetRef:
    kind: MeshSubset
    tags:
      app: hello-world
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          app.kubernetes.io/name: gateway
          k8s.kuma.io/namespace: kong
      default:
        action: Allow

Deploy a test application with TLS Using Gateway API

  • Deploy a simple "Hello World" application and configure it with Gateway API and Kong using TLS

Label default namespace for sidecar injection

kubectl label namespace default kuma.io/sidecar-injection=enabled

Deploy the manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-tls
  labels:
    app: hello-world-tls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-tls
  template:
    metadata:
      labels:
        app: hello-world-tls
    spec:
      containers:
      - name: hello-world-tls
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world-tls
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
spec:
  selector:
    app: hello-world-tls
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: hello-world-tls-gateway
  annotations:
    cert-manager.io/issuer: dns-issuer
spec:
  gatewayClassName: kong
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    # Replace with your value
    hostname: "testtls.alpacaislands.nl"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: hello-world-tls-cert
        namespace: default
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: hello-world-tls-route
spec:
  parentRefs:
  - name: hello-world-tls-gateway
  hostnames:
  # Replace with your value
  - "testtls.alpacaislands.nl"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: hello-world-tls
      port: 80

Allow traffic from gateway to the service

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  namespace: default
  name: hello-world-tls-permission
spec:
  targetRef:
    kind: MeshSubset
    tags:
      app: hello-world-tls
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          app.kubernetes.io/name: gateway
          k8s.kuma.io/namespace: kong
      default:
        action: Allow
⚠️ **GitHub.com Fallback** ⚠️