23 ‐ Kubernetes Priority Class - CloudScope/DevOpsWithCloudScope GitHub Wiki

Kubernetes Priority Class: Overview

In Kubernetes, Priority Classes are used to control the priority of Pods during scheduling and resource allocation. Pods with higher priority can preempt (evict) Pods with lower priority when resources are scarce, ensuring that critical workloads are scheduled before less important ones.

Key Concepts

  1. Priority Class: A way to assign a priority to Pods based on their criticality. Priority classes determine which Pods are more important and should be scheduled first, especially when resource contention occurs.

  2. Preemption: If a high-priority Pod cannot be scheduled due to resource constraints, it can preempt lower-priority Pods. This means that the lower-priority Pods may be evicted to free up resources for the higher-priority ones.

  3. Pod Priority: Every Pod is associated with a PriorityClass, which defines the priority of the Pod. The Pod’s priority value is used to decide whether a Pod can preempt other Pods or if it should be evicted.

How Priority Classes Work

  1. Priority Values:

    • The priority value is an integer, where higher values indicate higher priority.
    • Pods with a higher priority value are scheduled before Pods with a lower priority value.
    • If two Pods have the same priority, Kubernetes uses other factors like node and pod affinity to make scheduling decisions.

  2. Preemption Behavior:

    • When a Pod with a higher priority is pending and cannot be scheduled due to resource constraints, it can trigger preemption.
    • If preemption occurs, Kubernetes will attempt to evict lower-priority Pods to make room for the higher-priority Pod.
    • Pod Preemption happens only if preemptionPolicy is set to PreemptLowerPriority.

  3. PodPriority vs. PriorityClass:

    • The PodPriority is a value associated with the Pod, derived from its PriorityClass.
    • A PriorityClass defines the behavior of the Pods within it, such as preemptionPolicy.

Creating and Using Priority Classes

  1. Define a Priority Class: You create a PriorityClass resource to define the priority of the Pods in your cluster.

    Example:

    apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This is a high priority class."
preemptionPolicy: PreemptLowerPriority
    • value: The priority value for this class.
    • globalDefault: A boolean flag that, when set to true, makes this the default priority for Pods that do not specify a priority class.
    • preemptionPolicy: Determines if preemption should occur for this class. Options are PreemptLowerPriority (default) or Never.

  2. Assign a Priority Class to a Pod: Once you create a PriorityClass, you can assign it to a Pod by setting the priorityClassName field.

    Example:

    apiVersion: v1
kind: Pod
metadata:
  name: my-high-priority-pod
spec:
  priorityClassName: high-priority

  3. Default Priority Class:

    • If no priorityClassName is specified for a Pod, it will use the default PriorityClass (if defined). If no default is set, the Pod’s priority will be zero.

Preemption Policies

  • PreemptLowerPriority (default): A Pod can preempt lower-priority Pods to schedule itself.
  • Never: Preemption is disabled for the Pod. It won't preempt any other Pods, even if they have lower priority.

Use Cases for Priority Classes

  1. Critical Workloads: Assign high priority to critical workloads (like databases, or core applications) to ensure they are always scheduled before non-essential workloads.
  2. Batch Jobs: For jobs with flexible scheduling, you can assign a lower priority, allowing them to be preempted if the cluster runs out of resources.
  3. Resource Contention: In a situation where the cluster experiences resource shortages, high-priority Pods can ensure their execution by preempting low-priority Pods.

Limitations and Considerations

  1. Preemption Overhead: Preemption can lead to Pods being killed and rescheduled, which might introduce additional overhead or delays.
  2. Resource Starvation: If there are too many high-priority Pods, lower-priority Pods may not get scheduled, potentially causing resource starvation.
  3. Impact on Resource Quotas: If Pods are evicted as part of preemption, it may impact the resources available for other workloads in the cluster.
  4. Default Priority: Kubernetes does not have a default PriorityClass until you explicitly set one. If you want all Pods to have a default priority, you should configure a default PriorityClass.

Best Practices

  • Use for Critical Services: Only use high priority for workloads that truly need to be prioritized (e.g., databases, API servers, etc.).
  • Set Preemption Policy Carefully: Be cautious when enabling preemption, as it can lead to undesired Pod evictions.
  • Combine with Resource Requests: Priorities are most effective when used in combination with resource requests and limits, ensuring that critical Pods get the resources they need.
  • Avoid Overloading with High Priority: Use high-priority settings sparingly to prevent overloading the scheduler and evicting lower-priority but still important workloads.