Cluster setup w kuma mesh - JensvandeWiel/k3s-cluster-setup GitHub Wiki

This guide handles installing k3s, helm, cert-manager, external-dns, traefik ingress and kuma mesh. And you will create a example nginx deployment.

1. Install k3s (Without Default Traefik Proxy)

To install k3s and disable the default Traefik proxy in one step:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -

The --disable=traefik flag ensures that the default Traefik proxy is not installed, allowing you to install and configure a custom version of Traefik.
After installation, export the KUBECONFIG environment variable to manage the cluster:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

2. Install Helm

Helm simplifies deploying applications on Kubernetes via charts.

Add the Helm repository and install Helm:

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

3. Install ExternalDNS with Cloudflare

ExternalDNS automates the management of DNS records based on your Kubernetes resources.

Add the Bitnami Helm repository:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Install ExternalDNS with the Cloudflare provider:

helm install external-dns bitnami/external-dns \
--create-namespace \
--namespace external-dns \
--set provider=cloudflare \
--set cloudflare.apiToken=<YOUR_CLOUDFLARE_API_TOKEN> \
--set policy=sync \
--set txtOwnerId=external-dns-k3s \
--set cloudflare.proxied=false \
--set dnsRecordsPerPage=500

Replace <YOUR_CLOUDFLARE_API_TOKEN> with your Cloudflare API token.

4. Install Cert-Manager

Cert-Manager automates the creation and renewal of TLS/SSL certificates.

Add the Jetstack Helm repository and install Cert-Manager:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

Create ClusterIssuers for Let’s Encrypt:

Staging Issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: traefik

Production Issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: traefik

Apply the ClusterIssuer configurations:

kubectl apply -f cluster-issuer-staging.yaml
kubectl apply -f cluster-issuer-production.yaml

5. Install Kuma Service Mesh

Kuma will handle east-west (service-to-service) traffic with strict mTLS. We will also use kuma CNI so that the sidecars dont need a InitContainer relying on root privileges.

Install Kuma using Helm:

helm repo add kuma https://kumahq.github.io/charts
helm repo update
helm install kuma kuma/kuma \
  --namespace kuma-system \
  --create-namespace \
  --set controlPlane.mode=standalone \
  --set dataPlane.kdsGlobal.enabled=false \
  --set cni.enabled=true \
  --set cni.chained=true \
  --set cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d \
  --set cni.binDir=/var/lib/rancher/k3s/data/cni \
  --set cni.confName=10-flannel.conflist

Enable mTLS in the mesh:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
      mode: STRICT

kubectl apply -f mesh.yaml

Allow all traffic in the mesh

You could also skip this and add more selective permissions when deploying the example app.

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: allow-all
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  from:
  - targetRef:
      kind: Mesh
    default:
      action: Allow

kubectl apply -f mesh-permission.yaml

6. Install Custom Traefik Proxy as Default Ingress

Add the Traefik Helm repository:

helm repo add traefik https://traefik.github.io/charts
helm repo update

Create namespace:

kubectl create namespace traefik
kubectl label namespace traefik kuma.io/sidecar-injection=enabled

Install Traefik:

Traefik acts as the (API) gateway for Kuma, this means that you need to make sure that traffic can be sent to services and that the ports exposing to the network are not included (80, 443). You also need to tell traefik to use the service ClusterIP instead of the Pod IP.

helm install traefik traefik/traefik \
  --namespace traefik \
  --set ports.web.port=80 \
  --set ports.websecure.port=443 \
  --set ingressClass.enabled=true \
  --set ingressClass.isDefaultClass=true \
  --set providers.kubernetesCRD.enabled=true \
  --set providers.kubernetesCRD.nativeLBByDefault=true \
  --set providers.kubernetesIngress.enabled=true \
  --set providers.kubernetesIngress.nativeLBByDefault=true \
  --set deployment.annotations."kuma\.io/gateway"=enabled \
  --set deployment.podLabels."kuma\.io/sidecar-injection"=enabled \
  --set deployment.podAnnotations."kuma\.io/gateway"=enabled

Verify Traefik installation:

kubectl get pods -n traefik

7. Deploy a Test Application with Ingress and TLS

Deploy a simple "Hello World" application and configure it with TLS using Cert-Manager and Traefik.

Create the hello-world.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
        kuma.io/sidecar-injection: enabled
    spec:
      containers:
      - name: hello-world
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  annotations:
    # This must be set in case that sidepod injection is not namespace wide:
    # https://kuma.io/docs/2.9.x/using-mesh/managing-ingress-traffic/delegated/
    ingress.kubernetes.io/service-upstream: "true"
    # Make sure that traefik references to the service ClusterIP instead of the Pod IP
    traefik.ingress.kubernetes.io/service.nativelb: "true"
spec:
  selector:
    app: hello-world
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-production"
    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
spec:
  rules:
  - host: <YOUR_DOMAIN>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80
  tls:
  - hosts:
    - <YOUR_DOMAIN>
    secretName: hello-world-tls

Replace <YOUR_DOMAIN> with your actual domain.
Apply the deployment:

kubectl apply -f hello-world.yaml

If you havent set mesh traffic permissions earlier you can do this now:

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  namespace: default
  name: hello-world-to-traefik
spec:
  targetRef:
    kind: MeshSubset
    tags:
      app: hello-world
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          app.kubernetes.io/name: traefik
          k8s.kuma.io/namespace: traefik
      default:
        action: Allow

This allows traffic only from traefik to the hello-world service

kubectl apply -f hello-world-to-traefik-permission.yaml

Visit your domain, you should be greeted by nginx hello world.