access to nodes - juancamilocc/virtual_resources GitHub Wiki

Customizing the Kubelet Service in Kubernetes Cluster

In this guide, you will learn how to customize the Kubelet service in a Kubernetes cluster. The Kubelete is responsible for ensuring that containers run as expected. Usually, cloud providers restrict direct access to the Kubelet configuration. However, you can access and modify the configuration by deploying a privileged pod on the node you wish to customize.

General workflow

This guide provides step-by-step instructions on how to locate and modify the Kubelet configuration file to optimize performance, memory usage, and stability.

Identify Node

First of all we need to identify the node name.

kubectl get nodes -o wide
# NAME                                           STATUS   ROLES    AGE     VERSION
# <ip>.<region>.compute.internal                 Ready    <none>   4d22h   v1.31.5-eks-5d632ec
# <ip>.<region>.compute.internal                 Ready    <none>   4d21h   v1.31.5-eks-5d632ec
# <ip>.<region>.compute.internal                 Ready    <none>   4d21h   v1.31.5-eks-5d632ec
# <ip>.<region>.compute.internal                 Ready    <none>   4d21h   v1.31.5-eks-5d632ec

Based on the node name, we will use the following privileged pod to access the node, referenced in the section nodeName:.

apiVersion: v1
kind: Pod
metadata:
  name: node-shell-inspector
  namespace: kube-system
spec:
  volumes:
    - name: kube-api-access-psvmd
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: shell
      image: docker.io/alpine:3.13
      command:
        - nsenter
      args:
        - '-t'
        - '1'
        - '-m'
        - '-u'
        - '-i'
        - '-n'
        - sleep
        - '14000'
      resources: {}
      volumeMounts:
        - name: kube-api-access-psvmd
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: <node_name>
  hostNetwork: true
  hostPID: true
  hostIPC: true
  securityContext: {}
  schedulerName: default-scheduler
  priorityClassName: system-node-critical
  priority: 2000001000
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

Deploy the pod, as follows.

kubectl apply -f node-shell-inspector.yaml
kubectl -n kube-system get pods
# .
# .
# .
# node-shell-inspector                        1/1     Running   0               10s
# .
# .
# .

Accessing the Kubelet Configuration

Once the pod is running, we will access to it and locate the Kubelet file configuration.

kubectl -n kube-system exec -it node-shell-inspector -- bash
[root@ip-node kubelet]# cd /etc/kubernetes/kubelet
[root@ip-node kubelet]# ls
# kubelet-config.json
[root@ip-node kubelet]# cat kubelet-config.json
# {
#   "kind": "KubeletConfiguration",
#   "apiVersion": "kubelet.config.k8s.io/v1beta1",
#   "address": "0.0.0.0",
#   "authentication": {
#     "anonymous": {
#       "enabled": false
#     },
#     "webhook": {
#       "cacheTTL": "2m0s",
#       "enabled": true
#     },
#     "x509": {
#       "clientCAFile": "/etc/kubernetes/pki/ca.crt"
#     }
#   },
#   "authorization": {
#     "mode": "Webhook",
#     "webhook": {
#       "cacheAuthorizedTTL": "5m0s",
#       "cacheUnauthorizedTTL": "30s"
#     }
#   },
#   "clusterDomain": "cluster.local",
#   "hairpinMode": "hairpin-veth",
#   "readOnlyPort": 0,
#   "cgroupDriver": "systemd",
#   "cgroupRoot": "/",
#   "featureGates": {
#     "RotateKubeletServerCertificate": true
#   },
#   "protectKernelDefaults": true,
#   "serializeImagePulls": false,
#   "serverTLSBootstrap": true,
#   "tlsCipherSuites": [
#     "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
#     "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
#     "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
#     "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
#     "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
#     "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
#     "TLS_RSA_WITH_AES_256_GCM_SHA384",
#     "TLS_RSA_WITH_AES_128_GCM_SHA256"
#   ],
#   "clusterDNS": [
#     "<ip_dns>"
#   ],
#   "evictionHard": {
#     "memory.available": "100Mi",
#     "nodefs.available": "10%",
#     "nodefs.inodesFree": "5%"
#   },
#   "kubeReserved": {
#     "cpu": "80m",
#     "ephemeral-storage": "1Gi",
#     "memory": "893Mi"
#   },
#   "providerID": "aws:///<region<zone>>/i-<id_number>",
#   "systemReservedCgroup": "/system",
#   "kubeReservedCgroup": "/runtime"
# }

Modifying the Kubelet Configuration

Once you have identified the kubelet file, you can modify it as you wish. I will explain some useful flags you can implement in your kubelet configuration according to your needs.

Performance Optimization

"eventBurst": 50,       # Maximum number of events that kubelete can handle simultaneously  
"eventRecordQPS": 10,   # Events registered limit per second, it avoids spam
"maxPods": 250,         # Maximum pods number
"registryBurst": 20,    # Simultaneous connections to the registry
"registryPullQPS": 10,  # Download speed of images
"podPidsLimit": 5000    # Avoid many processes on the node

Memory and CPU Management

"cpuCFSQuota": true,            # Enable cpu limit in containers
  "cpuCFSQuotaPeriod": "100ms", # Set control period of cpu quota limit 
  "memorySwap": {               
    "swapBehavior": "Limited"   # Enable swap memory use to avoid OOM (Out Of Memory)
  }

Availability and Stability

"nodeStatusUpdateFrequency": "10s", # Send updates to the API server every 10s
"nodeLeaseDurationSeconds": 40,     # Time a node is mark as active without send heartbeat
"imageMinimumGCAge": "10m",         # Keep unused images for 10 minutes before deleting them
"imageGCHighThresholdPercent": 85,  # Collect images when the disk use reaches 85%
"imageGCLowThresholdPercent": 75    # Delete images until the disk use reaches 70%

NOTE: It is important to highlight that the above values are only a recommendation and you can adapt or change them according to your needs.

Restarting the Kubelet Service

After modifying the configuration, let's restart the Kubelet service.

sudo systemctl daemon-reload  
sudo systemctl restart kubelet

Verify the Kubelet service is running correctly.

[root@ip-node kubelet]# sudo systemctl status kubelet
# ● kubelet.service - Kubernetes Kubelet
#    Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
#   Drop-In: /etc/systemd/system/kubelet.service.d
#            └─10-kubelet-args.conf, 30-kubelet-extra-args.conf
#    Active: active (running) since Tue 2025-03-25 23:29:05 UTC; 4 days ago
#      Docs: https://github.com/kubernetes/kubernetes
#   Process: 3107 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5 (code=exited, status=0/SUCCESS)
#  Main PID: 3116 (kubelet)
#     Tasks: 17
#    Memory: 163.3M
#    CGroup: /runtime.slice/kubelet.service
#            └─3116 /usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint unix:///run/containerd/contai...
# .
# .
# .

Conclusions

Customizing the Kubelet service allows you to fine-tune the performance, resource allocation, and stability of your Kubernetes nodes. While cloud providers typically restrict direct access to these configurations, deploying a privileged pod enables necessary modifications.

Carefully consider the changes you make, as incorrect configurations can lead to node instability. Always test modifications in a controlled environment before applying them in production.

⚠️ **GitHub.com Fallback** ⚠️