29 ‐ Kubernetes Admission Controllers - CloudScope/DevOpsWithCloudScope GitHub Wiki

Admission Controllers in Kubernetes are a set of plugins that govern and enforce policies on objects during their lifecycle. They intercept API requests to the Kubernetes API server after the request is authenticated and authorized, but before the object is persisted in etcd. Admission controllers can either allow or deny requests, mutate resources, or trigger other actions.

1. Admission Control Flow:

   • Client Request → API Server → Admission Controllers → Persistent Store (etcd)
   • The admission process happens after authentication and authorization but before the object is saved in etcd.

2. Types of Admission Controllers:

   • Validating Admission Controllers: These only validate a request, ensuring that it conforms to required policies. They cannot modify the request, only reject it.
   • Mutating Admission Controllers: These can modify the request before it is persisted. They can modify resource objects or add annotations, labels, etc., based on the configuration.

3. Order of Execution:

   • Admission controllers are executed in the order they are specified in the Kubernetes API server configuration.
   • Mutating controllers are executed before Validating controllers.

4. Admission Control Lifecycle:

   1. Authentication: Verifies the identity of the user.
   2. Authorization: Checks whether the user has the necessary permissions to perform the operation.
   3. Admission Control: Validates or mutates the request before the object is persisted.

5. Admission Control Phases:

   • Pre-Admission: The request is processed by authentication and authorization.
   • Mutating: Mutating controllers can change the resource.
   • Validating: Validating controllers check for compliance and reject non-conforming requests.

Here’s a list of commonly used admission controllers:

• NamespaceLifecycle: Ensures resources are not created in deleted namespaces.
• LimitRanger: Enforces resource limits (CPU, memory) on objects like Pods.
• ServiceAccount: Automatically adds service account information to Pods.
• PodSecurityPolicy (deprecated in 1.25, replaced by PodSecurity): Controls Pod security policies to limit security risks.
• DefaultTolerationSeconds: Adds default tolerations to Pods, based on configuration.
• DefaultStorageClass: Automatically assigns a storage class to persistent volume claims (PVCs) when none is specified.

• AlwaysAdmit: Always allows all requests.
• DenyEscalatingExec: Prevents users from escalating their privileges via kubectl exec.
• PodSecurity: Enforces Pod security standards (replacing the deprecated PodSecurityPolicy).
• ResourceQuota: Ensures that quotas are respected within a namespace.
• ValidatingAdmissionWebhook: Allows for external webhooks to perform custom validation.

1. Security Policies: You can enforce policies around Pod security, e.g., disallowing privileged containers or enforcing specific security contexts.
2. Resource Limits: Automatically enforce CPU and memory resource limits on Pods.
3. Namespace Validation: Ensure that resources are created only in the valid namespaces.
4. Custom Validation Logic: Use webhooks for custom validation beyond the built-in controllers.
5. Mutating Resources: Automatically inject environment variables, labels, or annotations into resources as they are created.

Admission controllers are configured in the Kubernetes API server by the --enable-admission-plugins flag (or --disable-admission-plugins for disabling specific ones). When configuring an admission controller, you can specify the controllers to be enabled or disabled.

Example API server startup command:

kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,PodSecurity,ValidatingAdmissionWebhook --disable-admission-plugins=PodSecurityPolicy

• MutatingAdmissionWebhook: Allows external systems to modify the request (e.g., add default values or labels to resources before they are persisted).
• ValidatingAdmissionWebhook: Allows external systems to validate a resource against custom business logic or policies.

Admission webhooks can be used to extend the native admission control logic by integrating third-party services or custom code.

A webhook might inspect a resource, validate it against a set of rules, and either reject the request or modify it before allowing it to be saved in etcd. Webhooks must implement HTTP endpoints and be registered with the API server.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-mutating-webhook
webhooks:
  - name: mutating.example.com
    clientConfig:
      url: "https://your-webhook-service"
    rules:
      - operations: ["CREATE"]
        apiGroups: ["apps"]
        apiVersions: ["v1"]
        resources: ["deployments"]

• Keep the order in mind: Mutating controllers should be placed before validating controllers in the API server configuration.
• Use webhooks for custom logic: When built-in controllers don’t meet your needs, consider writing a webhook for specific business logic.
• Do not block on heavy operations: Admission controllers should perform fast checks to avoid delaying the Kubernetes API server.
• Ensure idempotency: Admission controllers should ensure that repeated actions (e.g., retries) do not cause unexpected results.
• Audit logs: Always monitor and log admissions for debugging and security auditing purposes.

• API server logs: Inspect logs of the Kubernetes API server to identify issues related to admission controllers.
• Admission controller logs: If using webhooks, check the logs of the webhook service for errors.
• kubectl describe: Use kubectl describe to get detailed information about admission decisions, especially when requests are rejected.

Example for Enforcing label in all deployment:

To deploy the mutating webhook service in Kubernetes, you will need to follow a few steps. These steps involve creating the webhook service, deploying it in Kubernetes, and ensuring that the Kubernetes API server can interact with it securely.

I'll walk you through deploying a simple mutating webhook service (like the example Python service I gave earlier) and exposing it in the Kubernetes cluster. We'll assume you're using a self-signed certificate for HTTPS communication, as Kubernetes requires webhooks to communicate over HTTPS.

First, ensure you have the webhook server code ready (e.g., Python Flask service). I'll use the same Python example from before, where the webhook will add a label (env: production) to a Deployment.

Here’s the sample Python webhook server code (mutating-webhook.py):

from flask import Flask, request, jsonify
import json

app = Flask(__name__)

@app.route('/mutate', methods=['POST'])
def mutate():
    admission_review = request.get_json()
    deployment = admission_review['request']['object']
    
    # Check for the "labels" annotation in the Deployment
    annotations = deployment.get('metadata', {}).get('annotations', {})
    
    # If the "add-labels" annotation exists, use it to add labels
    label_str = annotations.get('add-labels', '')
    
    # If the annotation exists and is not empty, add it as labels
    if label_str:
        labels = deployment.setdefault('metadata', {}).setdefault('labels', {})
        
        # Parse the "add-labels" annotation as a comma-separated string (e.g., "env:production,team:dev")
        label_pairs = label_str.split(',')
        for label in label_pairs:
            key, value = label.split(':')
            labels[key] = value
    
    # Prepare the response
    admission_response = {
        "response": {
            "uid": admission_review['request']['uid'],
            "allowed": True,
            "patchType": "JSONPatch",
            "patch": json.dumps([{
                "op": "add",
                "path": "/metadata/labels",
                "value": labels
            }]).encode('utf-8').decode('utf-8')
        }
    }

    return jsonify(admission_response)

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=443)

Next, we need to containerize the Python webhook service so that it can run inside a Kubernetes pod.

Create a Dockerfile to build the container image:

FROM python:3.9-slim

WORKDIR /app
COPY mutating-webhook.py /app

RUN pip install flask

CMD ["python", "mutating-webhook.py"]

docker build -t mutating-webhook:latest .

After building the image, push it to a container registry (e.g., Docker Hub, Google Container Registry, or Amazon ECR).

docker tag mutating-webhook:latest <your-registry>/mutating-webhook:latest
docker push <your-registry>/mutating-webhook:latest

Once the image is in the registry, we can now deploy it in Kubernetes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mutating-webhook
  labels:
    app: mutating-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mutating-webhook
  template:
    metadata:
      labels:
        app: mutating-webhook
    spec:
      containers:
      - name: mutating-webhook
        image: <your-registry>/mutating-webhook:latest
        ports:
        - containerPort: 443
        volumeMounts:
        - mountPath: /etc/webhook/certs
          name: webhook-certs
          readOnly: true
      volumes:
      - name: webhook-certs
        secret:
          secretName: webhook-server-cert

This deployment runs the Python Flask application inside a Kubernetes pod.

apiVersion: v1
kind: Service
metadata:
  name: mutating-webhook-service
spec:
  ports:
    - port: 443
      targetPort: 443
  selector:
    app: mutating-webhook

This service exposes the mutating webhook on port 443 within the cluster.

Kubernetes requires webhooks to be served over HTTPS, so we need to create a TLS certificate for the webhook server.

First, generate a self-signed certificate (if you don't already have one):

openssl req -x509 -newkey rsa:4096 -keyout webhook-server.key -out webhook-server.crt -days 365 -nodes -subj "/CN=mutating-webhook-service.default.svc"

Then, create a Kubernetes secret from the certificate and key:

kubectl create secret tls webhook-server-cert --cert=webhook-server.crt --key=webhook-server.key -n default

kubectl apply -f mutating-webhook-deployment.yaml
kubectl apply -f mutating-webhook-service.yaml
kubectl apply -f webhook-server-cert.yaml

Now that the webhook service is deployed and exposed via the Kubernetes service, you need to create the MutatingWebhookConfiguration that tells Kubernetes to use this webhook for certain operations (in this case, for CREATE operations on Deployments).

Here’s the MutatingWebhookConfiguration YAML (mutating-webhook-configuration.yaml):

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: add-label-to-deployments
webhooks:
  - name: mutate.deployments.k8s.io
    clientConfig:
      service:
        name: mutating-webhook-service
        namespace: default
        path: /mutate
      caBundle: <CA-BUNDLE>  # Base64-encoded CA certificate used by your TLS secret (the webhook server's certificate)
    rules:
      - operations: ["CREATE"]
        apiGroups: ["apps"]
        apiVersions: ["v1"]
        resources: ["deployments"]
    admissionReviewVersions: ["v1"]
    sideEffects: None

• caBundle: This is the base64-encoded certificate authority (CA) certificate that Kubernetes will use to validate the webhook’s server certificate. You can get the certificate from your Kubernetes secret and base64-encode it.

To extract the CA certificate from the Kubernetes secret:

kubectl get secret webhook-server-cert -n default -o jsonpath='{.data.cert\.crt}' | base64 --decode > ca.crt

Then, base64-encode the ca.crt:

cat ca.crt | base64 -w 0

Paste the resulting base64 string into the caBundle field of the MutatingWebhookConfiguration.

kubectl apply -f mutating-webhook-configuration.yaml

Create a test Deployment to check if the webhook is working and automatically adding the env: production label.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  annotations:
    add-labels: "env:production,team:dev"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest

Create the Deployment:

kubectl apply -f nginx-deployment.yaml

After the deployment is created, check that the env: production label is added:

kubectl get deployment nginx-deployment -o yaml

You should see the label env: production added under the metadata.labels field.

To clean up after testing, you can delete the resources:

kubectl delete mutatingwebhookconfiguration add-label-to-deployments
kubectl delete service mutating-webhook-service -n default
kubectl delete deployment mutating-webhook -n default
kubectl delete secret webhook-server-cert -n default

In this process, we:

1. Created a Python-based webhook service that adds a label to Deployments.
2. Dockerized the service and pushed it to a container registry.
3. Deployed the service in Kubernetes using a Deployment and exposed it via a Service.
4. Created a TLS secret for secure communication.
5. Configured Kubernetes to call this webhook using a MutatingWebhookConfiguration.
6. Tested the webhook by creating a Deployment, and verifying the label was added automatically.

This is a simple example, and in a production environment, you'd want to use proper certificates, ensure high availability of your webhook service, and follow best practices for securing your webhooks.