Cluster setup w kuma mesh and kong gateway - JensvandeWiel/k3s-cluster-setup GitHub Wiki
This guide handles installing k3s, helm, cert-manager, external-dns, Kong ingress/gateway and kuma mesh.
- To install k3s and disable the default Traefik proxy in one step:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -
- The
--disable=traefik
flag ensures that the default Traefik proxy is not installed, allowing you to install and configure a custom version of Traefik. - After installation, export the
KUBECONFIG
environment variable to manage the cluster:
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
- Helm simplifies deploying applications on Kubernetes via charts.
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
- Kuma will handle east-west (service-to-service) traffic with strict mTLS. We will also use kuma CNI so that the sidecars don't need a InitContainer relying on root privileges.
helm repo add kuma https://kumahq.github.io/charts
helm repo update
helm install kuma kuma/kuma \
--namespace kuma-system \
--create-namespace \
--set controlPlane.mode=standalone \
--set dataPlane.kdsGlobal.enabled=false \
--set cni.enabled=true \
--set cni.chained=true \
--set cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d \
--set cni.binDir=/var/lib/rancher/k3s/data/cni \
--set cni.confName=10-flannel.conflist
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
name: default
spec:
mtls:
enabledBackend: ca-1
backends:
- name: ca-1
type: builtin
mode: STRICT
- You could also skip this and add more selective permissions when deploying the example app.
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
name: allow-all
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
from:
- targetRef:
kind: Mesh
default:
action: Allow
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: kong
annotations:
konghq.com/gatewayclass-unmanaged: 'true'
spec:
controllerName: konghq.com/kic-gateway-controller
helm repo add kong https://charts.konghq.com
helm repo update
helm install kong kong/ingress -n kong --create-namespace
kubectl label namespace kong kuma.io/sidecar-injection=enabled
kubectl rollout restart -n kong deployment kong-gateway kong-controller
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.8.0
- ExternalDNS automates the management of DNS records based on your Kubernetes resources.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install external-dns bitnami/external-dns \
--create-namespace \
--namespace external-dns \
--set provider=cloudflare \
--set cloudflare.apiToken=<YOUR_CLOUDFLARE_API_TOKEN> \
--set policy=sync \
--set txtOwnerId=external-dns-k3s \
--set cloudflare.proxied=false \
--set dnsRecordsPerPage=500 \
--set sources[0]=service \
--set sources[1]=ingress \
--set sources[2]=gateway-httproute \
--set sources[3]=gateway-grpcroute \
--set sources[4]=gateway-tcproute \
--set sources[5]=gateway-tlsroute \
--set sources[6]=gateway-udproute
- Replace
<YOUR_CLOUDFLARE_API_TOKEN>
with your Cloudflare API token.
- Cert-Manager automates the creation and renewal of TLS/SSL certificates.
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true \
--set "extraArgs={--enable-gateway-api}"
- You can choose between issuers, we will show how to create a staging cluster issuer, production issuer and a normal DNS issuer in the default namespace. You can choose between any of these, or use multiple.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Replace with your value
email: <YOUR_EMAIL>
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: kong
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
# Replace with your value
email: <YOUR_EMAIL>
privateKeySecretRef:
name: letsencrypt-production
solvers:
- http01:
ingress:
class: kong
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: dns-issuer
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
# Replace with your value
email: <YOUR_EMAIL>
privateKeySecretRef:
name: dns-issuer
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
- Deploy a simple "Hello World" application and configure it with Gateway API and Kong
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
labels:
app: hello-world
kuma.io/sidecar-injection: enabled
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
kuma.io/sidecar-injection: enabled
spec:
containers:
- name: hello-world
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
annotations:
ingress.kubernetes.io/service-upstream: "true"
spec:
selector:
app: hello-world
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: hello-world-gateway
spec:
gatewayClassName: kong
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hello-world-route
spec:
parentRefs:
- name: hello-world-gateway
hostnames:
# Replace with your value
- "test.alpacaislands.nl"
rules:
- backendRefs:
- name: hello-world
port: 80
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
namespace: default
name: hello-world-permission
spec:
targetRef:
kind: MeshSubset
tags:
app: hello-world
from:
- targetRef:
kind: MeshSubset
tags:
app.kubernetes.io/name: gateway
k8s.kuma.io/namespace: kong
default:
action: Allow
- Deploy a simple "Hello World" application and configure it with Gateway API and Kong using TLS
kubectl label namespace default kuma.io/sidecar-injection=enabled
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-tls
labels:
app: hello-world-tls
spec:
replicas: 1
selector:
matchLabels:
app: hello-world-tls
template:
metadata:
labels:
app: hello-world-tls
spec:
containers:
- name: hello-world-tls
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: hello-world-tls
annotations:
ingress.kubernetes.io/service-upstream: "true"
spec:
selector:
app: hello-world-tls
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: hello-world-tls-gateway
annotations:
cert-manager.io/issuer: dns-issuer
spec:
gatewayClassName: kong
listeners:
- name: https
protocol: HTTPS
port: 443
# Replace with your value
hostname: "testtls.alpacaislands.nl"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: hello-world-tls-cert
namespace: default
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hello-world-tls-route
spec:
parentRefs:
- name: hello-world-tls-gateway
hostnames:
# Replace with your value
- "testtls.alpacaislands.nl"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: hello-world-tls
port: 80
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
namespace: default
name: hello-world-tls-permission
spec:
targetRef:
kind: MeshSubset
tags:
app: hello-world-tls
from:
- targetRef:
kind: MeshSubset
tags:
app.kubernetes.io/name: gateway
k8s.kuma.io/namespace: kong
default:
action: Allow