22 ‐ POD Scheduling - CloudScope/DevOpsWithCloudScope GitHub Wiki

Taints and tolerations and node affinity are both mechanisms in Kubernetes that allow you to control where pods are scheduled within a cluster, but they serve different purposes and operate in distinct ways.

1. Taints and Tolerations:

  • Taints are applied to nodes. They allow nodes to repel certain pods, except the pods that have the corresponding tolerations.

  • Tolerations are applied to pods. They allow the pod to be scheduled on nodes that have specific taints, essentially telling Kubernetes, "This pod can tolerate the taint on this node."

How it works:

  • When a taint is applied to a node (e.g., key=value:NoSchedule), the node is considered to "repel" any pods unless those pods have the matching toleration (e.g., key=value).
  • If a pod has the right toleration, it will be able to run on the node even if the node has a taint.

Example:

  • A node might have a taint key=heavy-load:NoSchedule to prevent lightweight pods from being scheduled there. However, a pod can tolerate this taint if it has the toleration key=heavy-load.

Use Case:

  • Taints and tolerations are typically used for cases like isolating certain workloads (e.g., heavy workloads on specific nodes) or ensuring specific pods are scheduled only on nodes that meet certain criteria.

2. Node Affinity:

  • Node affinity is a pod specification used to influence which nodes a pod should or should not be scheduled on, based on the labels on those nodes.

  • Node affinity allows you to set rules for pod scheduling based on node labels and operators, similar to node selectors, but it is more flexible and powerful.

How it works:

  • Node affinity rules are defined in the pod specification, and Kubernetes tries to schedule the pod only on nodes that meet the defined criteria.
  • There are two types of node affinity:
    1. RequiredDuringSchedulingIgnoredDuringExecution: This is a hard requirement — the pod will not be scheduled unless the node matches the affinity rule.
    2. PreferredDuringSchedulingIgnoredDuringExecution: This is a soft preference — Kubernetes will try to schedule the pod on a node that satisfies the affinity, but it is not mandatory.

Example:

  • You could specify that a pod should be scheduled on nodes with a label disk=ssd, or on nodes in a specific zone using the topology.kubernetes.io/zone label.

Use Case:

  • Node affinity is used when you need to control pod placement based on node characteristics like hardware type, geographic location, or custom labels you apply to nodes.

Key Differences:

Aspect Taints and Tolerations Node Affinity
Applied to Nodes (taints) and Pods (tolerations) Pods (with node affinity rules)
Purpose Prevents pods from being scheduled on certain nodes Controls pod scheduling based on node labels
Operation Nodes "repel" pods, unless pods have matching tolerations Pods are scheduled on nodes matching affinity rules
Granularity Can apply "NoSchedule", "PreferNoSchedule", or "NoExecute" taints Can be required or preferred based on node labels
Example Use Case Isolate workloads, e.g., high-load nodes for heavy workloads Ensure pods run on specific hardware types, zones, or other node attributes

Summary:

  • Taints and tolerations control whether a pod can be scheduled on a specific node based on the node’s characteristics (taints). It's a way to repel certain pods from nodes unless they can tolerate the taints.
  • Node affinity controls where a pod can be scheduled by specifying conditions related to the node labels. It is a way to express affinity or preference for particular node attributes, like hardware or location.

These two mechanisms can also be used together. For example, you might use node affinity to ensure a pod is scheduled on a specific node type (like SSD nodes), while using taints and tolerations to ensure that only specific types of pods can be scheduled there.

In Kubernetes, node selectors and node labels are closely related concepts used to influence the scheduling of pods onto nodes in a cluster. Let's break down what node labels are and how node selectors work.

1. Node Labels:

  • Labels in Kubernetes are key-value pairs that are attached to objects like nodes, pods, and other resources. Labels are intended to convey identifying attributes of these resources.

  • Node labels specifically are key-value pairs attached to nodes in the cluster, providing metadata that can be used to describe characteristics of the node. For example, labels can describe the type of machine, its geographic location, or any other attributes relevant to scheduling or selection.

Example of a Node Label:

kubectl label nodes node1 region=us-east

In this example, the label region=us-east is applied to the node node1.

Common use cases for node labels:

  • Specifying the type of hardware available on the node, e.g., gpu=true, ssd=true.
  • Indicating the geographical location of the node, e.g., zone=us-east-1a.
  • Defining custom attributes, like tier=frontend or environment=production.

2. Node Selector:

  • Node Selector is a way to constrain pod scheduling based on the labels on nodes. It's a field in the pod specification that specifies the labels a node must have for a pod to be scheduled there.

  • In simple terms, a node selector ensures that a pod can only be scheduled on nodes that have specific labels. The node selector is a basic form of node affinity, allowing you to filter nodes based on their labels.

Example of Using Node Selector:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  nodeSelector:
    region: us-east

In this example, the pod my-pod can only be scheduled on nodes that have the label region=us-east. If no nodes with that label are available, the pod will not be scheduled.

Differences Between Node Selector and Node Affinity:

While node selector is simple and works with exact label matches, node affinity provides more flexibility and is preferred for more complex scheduling needs. Here's a comparison:

Aspect Node Selector Node Affinity
Function Select nodes based on exact label matches Select nodes based on label matches and rules
Complexity Simple, only allows key-value pairs More flexible, allows rules like In, NotIn, Exists
Scheduling Type Only exact matches of node labels Supports both required and preferred rules
Usage Basic pod-to-node matching More complex pod-to-node matching with preference
Example Usage nodeSelector: {region: us-east} affinity: {nodeAffinity: {requiredDuringSchedulingIgnoredDuringExecution: {nodeSelectorTerms: [{matchExpressions: [{key: region, operator: In, values: [us-east]}]}]}}}

Example Combining Labels, Node Selectors, and Node Affinity:

  1. Labeling Nodes:
kubectl label nodes node1 region=us-east disk=ssd
kubectl label nodes node2 region=us-west disk=hdd
  1. Pod with Node Selector:
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  nodeSelector:
    region: us-east

This pod will only be scheduled on node1 because it has the label region=us-east.

  1. Pod with Node Affinity:
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disk
            operator: In
            values:
            - ssd

In this case, the pod will only be scheduled on nodes that have the label disk=ssd, and this is enforced using node affinity rules.

Summary:

  • Node Labels are key-value pairs assigned to nodes that describe certain attributes of the node.
  • Node Selector is a simple mechanism used to schedule pods based on specific labels on nodes.
  • Node Affinity is a more powerful and flexible method of controlling pod scheduling based on node labels, supporting complex rules and preferences.

Both node selectors and node affinity leverage node labels, but node affinity is typically used for more advanced and nuanced scheduling requirements.