Kubernetes Scheduling: Affinity, Taints, and Topology

An unstable cluster during a peak traffic event often stems from poor workload placement policies. When replica pods crowd a single node, a small hardware failure can cause a major application outage.

Kubernetes scheduling matches new pods to the right nodes in a cluster based on resource requests and custom placement rules. The system uses the kube-scheduler to filter out nodes without enough room, and then ranks the remaining hosts to find the best fit. Platform teams use tools like node affinity, pod affinity, taints, tolerations, and topology spread constraints to group workloads or spread replicas across zones. These options let you keep related services close together, reserve dedicated hosts for special tasks, or spread replicas across different availability zones. Setting these policies correctly prevents hardware strain, keeps critical applications online, and lets teams manage large cluster fleets with very little effort.

Schedule a free demo to see how Plural handles kubernetes scheduling and fleet placement for you.

To help you build highly available cluster fleets, this guide explains exactly how the system makes its decisions. We will start by showing how the scheduler works, looking first at How the Kubernetes Scheduler Picks a Node. The path begins with

How the Kubernetes Scheduler Picks a Node

Kubernetes scheduling is how the control plane decides where to place your workloads. When you create a pod, it enters a pending state. The kube-scheduler must find the best node for this pod to run on. As shown in the Kubernetes scheduler guide, this process takes into account pod resource needs, policy constraints, and affinity rules.

First, the scheduler pulls pending pods from a scheduling queue. The scheduler then runs them through a pipeline to find the best host. This work has two main phases called filtering and scoring.

The filtering phase and node predicates

In the filtering phase, the scheduler finds all nodes where the pod can actually run. The scheduler runs a set of checks called predicates on each node in the cluster. These checks filter out nodes that do not meet the pod needs.

For example, a check might look at node RAM and CPU limits. If a node does not have enough space, the filter removes it. Other predicates look at things like node ports, taints, or selector matches.

The scheduler checks every node against these rules. If no node passes, the pod cannot be placed. The other hosts that pass all checks are called feasible nodes.

The scoring phase and priority functions

Once the filter phase is done, the scheduler must rank the remaining feasible nodes. This ranking happens in the scoring phase. The scheduler runs a set of priority functions to give each node a score from zero to ten.

These functions look at rules like node affinity, pod spread, and kubernetes scheduling priority and preemption policies. The node with the highest total score is the winner. If two nodes have the same top score, the scheduler picks one at random.

The scoring phase ensures that workloads land on the best host. Some scores favor nodes that already have the required container images. Others favor nodes that balance the resource load across the whole cluster.

The final binding process

The final phase of the pipeline is called binding. Once the best node is chosen, the scheduler creates a binding object. This object tells the API server to assign the pod to the chosen node.

If no node passes the filter stage, the pod is marked as unschedulable. It goes back to the scheduling queue to wait. The scheduler will try to place it again when resources change or new nodes join.

Once bound, the local agent on the node takes over. This agent, called the kubelet, gets the pod spec and starts the containers. The scheduler is now free to process the next pending pod in the queue.

How Does Node Affinity Control Where Your Pods Run?

Kubernetes uses labels on nodes to guide pod placement. Node affinity is a key feature of kubernetes scheduling and autoscaling that lets you choose which nodes can run your pods. The scheduler checks these rules when it filters the list of nodes.

Understanding Node Affinity Rules

Nodes must have labels before you can use affinity. Cluster operators can add custom labels like storage type or team names. The scheduler reads these labels on each node during its filtering step. If a node does not have the right label, the scheduler rules it out.

You will often use the In and NotIn operators, but you can also use Exists and DoesNotExist to check for label presence. For number checks, the scheduler supports Lt and Gt. These operators check if a label value is less than or greater than a number. This lets you write precise rules based on node properties.

Required vs Preferred Rules

You can make your node rules hard or soft. Hard rules use the requiredDuringSchedulingIgnoredDuringExecution setting. If the cluster does not have a node that matches a hard rule, the pod will not schedule.

Soft rules use the preferredDuringSchedulingIgnoredDuringExecution setting. In this case, the scheduler tries to find a match but will still place the pod on an unmatched node if needed. For instance, teams at New York University use these settings to direct workloads to specific hardware pools.

Hard rules can lead to scheduling failures if space is tight. If you run out of nodes with the right labels, your pods will stay in a pending state. On the other hand, soft rules offer more leeway. They allow pods to run even when the ideal nodes are full, which keeps your apps online.

The YAML snippet below shows a basic node affinity setup. This spec tells the scheduler that the pod must run in the us-east-1a zone. The scheduler will ignore nodes in other zones during the filtering stage.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1a

How Node Anti-Affinity Works

You can also use node affinity to keep pods apart. By using the NotIn or DoesNotExist operators, you create anti-affinity. This prevents the scheduler from placing pods on nodes that carry certain labels, which keeps your workloads apart.

Node anti-affinity is crucial when you want to spread replicas of a service across different hardware. This setup helps avoid single points of failure. For example, you can prevent two database replicas from landing on the same host or rack.

These placement rules play a major role in kubernetes scheduling scoring. During the scoring phase, the scheduler rates nodes based on your preferred rules. A node that matches more rules receives a higher score, making it the top choice for pod placement.

When Should You Use Pod Affinity and Anti-Affinity?

Node affinity helps you choose nodes based on node labels. In contrast, pod affinity and anti-affinity are core parts of kubernetes scheduling that let you set rules based on other running pods. This approach is highly useful when you need to co-locate workloads or spread them out for safety.

Why Schedule Based on Existing Pods?

Sometimes your app services need to talk to each other with low latency. For example, you can use pod affinity to schedule a web app on the same node as its cache. This setup keeps network traffic local and speeds up your response times. The scheduler reads these rules using labels on existing pods and places the new pod on a good node.

On the other hand, pod anti-affinity does the opposite by spreading your workloads. Use it to prevent two replicas of a critical service from running on the same node or in the same zone. If a node fails, your other replica stays online. This setup is a pillar of high uptime in production fleets. Research into scheduler design shows that precise placement rules prevent resource starvation and keep services online (PMC10058403).

How Hard and Soft Constraints Work

Kubernetes divides these rules into hard constraints and soft rules. Hard constraints use the requiredDuringSchedulingIgnoredDuringExecution field, which is sometimes called required or requiredAggregatedRetry in custom scheduling setups. If the cluster cannot find a node that meets a hard rule, the pod remains unscheduled. This is a common pain point when setting up kubernetes scheduling for virtual kubelets because resources can be tight.

Soft rules use the preferredDuringSchedulingIgnoredDuringExecution field. This tells the scheduler to try its best to meet your wishes. If no node fits, the scheduler will still bind the pod to a node that does not match. Soft rules are perfect when you want to spread workloads but prefer a running pod over a pending one.

Both types require a topologyKey. This key defines the domain, such as a node or a zone, where the rules apply. For example, if you set the key to kubernetes.io/hostname, the scheduler looks at single nodes. If you use topology.kubernetes.io/zone, it looks at entire zones.

A Practical Pod Anti-Affinity Configuration

You can set these rules directly in your pod spec. Below is a YAML snippet that shows how to configure anti-affinity. It prevents the scheduler from placing two replicas of a web server on the same node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: nginx:1.25

This config keeps your app safe from single node failures. You can combine these settings with kubernetes scheduling with tolerations to build secure systems.

How Do Taints and Tolerations Reserve Nodes for Special Workloads?

Kubernetes uses a system of taints and tolerations to restrict which pods can run on certain nodes. These tools help you assign workloads to the right nodes for node scheduling needs. A taint on a node tells the system to repel certain pods. The scheduler will not place a pod on that node unless the pod has a matching toleration. This control is a key part of kubernetes scheduling in large fleets.

Core concepts and effects

Each taint has a key, a value, and an effect. The key and value are simple labels, while the effect tells the system what to do. The system supports three taint effects to handle pod placement. These effects are NoSchedule, PreferNoSchedule, and NoExecute. You set these effects in your node config.

NoSchedule blocks new pods from running on the node. PreferNoSchedule is a soft rule where the system tries to avoid the node if possible. NoExecute works in a new way because it evicts any running pods that do not have a matching toleration. This allows you to remove workloads when a node fails.

Comparing toleration operators

To allow a pod to run on a tainted node, you must define matching rules when you set up your kubernetes scheduling with tolerations. These rules use operators like Equal or Exists to match node taints. The Equal operator needs both a key and an exact value to match. The Exists operator is simpler and matches any taint value for a given key. The system uses the Equal operator if you do not choose one.

When using NoExecute, you can add a timer called tolerationSeconds to your pods. If a node gets a new taint, this timer tells the pod how long to stay. For instance, a pod can stay for thirty minutes before the system evicts it. This delay is helpful for brief outages, giving nodes time to recover.

Production use cases in enterprise environments

Taints are vital for keeping nodes apart in large production clusters. You can reserve nodes with fast GPU hardware for machine learning jobs. You can also reserve control-plane nodes to run only system software, which keeps user apps off key servers. Finally, taints help isolate data in regulated or air-gapped workloads, letting banks and clinics meet strict safety rules without losing speed.

Finally, certain system tasks must run on every node, even with taints. These services run as DaemonSets, and the DaemonSet controller auto-adds tolerations for common node issues. This covers problems like disk pressure or when a node is not ready yet. By doing this, your critical logs and metrics can still run on every server.

How Do Topology Spread Constraints Keep Your Fleet Available?

High availability in a cluster needs smart pod placement. Using topology spread constraints helps you spread pods across failure domains. This method controls how pods spread across regions, zones, and nodes. It works with the scheduler to keep your workloads safe from zone outages. This setup helps teams balance work across nodes during kubernetes scheduling and autoscaling events.

Understanding topology domains and failure boundaries

Failure domains are groups of nodes in your cluster. In public clouds like AWS, GCP, or Azure, these domains map to regions and zones. The scheduler uses node labels like topology.kubernetes.io/zone to find these zones. Spreading pods across zones makes sure that a single power cut does not take down your entire app.

This approach is better than simple anti-affinity rules. While anti-affinity is a hard yes-or-no choice, spread constraints let you set a soft balance. It lets you run multiple replicas on the same node but still keeps them as even as possible. You get fine control over your cluster without blocking new pod starts.

Key parameters for configuring even pod distribution

To set up these rules, you will use a few key fields. The maxSkew field sets the highest gap in pod counts between any two domains. For instance, a skew of one means one zone can only have one more pod than another zone. The topologyKey tells the system which node label defines the domain.

You must also set the whenUnsatisfiable field. If you choose DoNotSchedule, the system will keep a pod pending if it cannot meet the spread rules. In tight clusters, this can lead to kubernetes scheduling priority and preemption, where high priority pods push out other workloads. If you choose ScheduleAnyway, the scheduler will still place the pod on the least skewed node it can find.

You can also use matchLabelKeys to filter which pods count toward the skew. This lets you count only pods from the same deployment. It stops old or other pods with the same labels from skewing your pod counts.

Deploying topology spread constraints in production

Deploying these rules requires a clear YAML configuration. You add the spread rules to your pod spec. Many teams use these rules to keep databases online across many zones. For instance, you can look at the University of Wisconsin MySQL Helm chart to see how database replicas are spread across zones.

The following example shows a basic configuration for spreading replicas evenly across availability zones:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: web-server

This simple block keeps your web servers balanced. It tells the scheduler to keep the skew at one across zones. If one zone fails, your other zones still run enough pods to handle your users.

What Kubernetes Scheduling Anti-Patterns Hurt Performance and Availability?

Kubernetes clusters often run into trouble when teams do not set up their pods and nodes well. The kube-scheduler decides node placement by running each pod through a two-step path. It first filters out nodes that do not fit, then scores the rest to find the best host. If you set your rules poorly, you can block this flow.

Common scheduling mistakes in production

Many teams start by using simple node selectors. While this works at first, it lacks the power of node affinity. As a cluster grows, node selectors become too rigid to manage.

Over time, you might also forget to set pod requests and limits. Without these values, the scheduler cannot filter nodes based on CPU or memory use. This leads to node overload and bad app speed.

You must plan for these pitfalls to keep your services online. For example, leaving node taints unmanaged will strand workloads on idle hosts. Set up matching kubernetes scheduling with tolerations to avoid this.

The cost of poor scheduling choices

Anti-pattern	Symptom	Recommended fix
Single-zone replicas	One zone outage takes down all app copies.	Use pod anti-affinity to spread replicas.
NodeSelector overuse	Pods get stuck if a specific node label is missing.	Use preferred node affinity for soft rules.
Missing requests or limits	Nodes get full, causing pod evictions.	Set CPU and memory limits on each pod.
Unmanaged node taints	Workloads cannot run on idle nodes.	Set up matching pod tolerations.
Ignoring topology spread	Stateful pods cluster on the same host or zone.	Set topology spread constraints with maxSkew.
Placement ties	Pods land on the same node during rapid scale up.	Use anti-affinity rules to guide scheduling scoring.

How the scheduler evaluates placement rules

To avoid these issues, you must know how the scheduler works. In the filter phase, it checks your pod requests against node capacity. If you skip requests, the filter step fails to screen out crowded hosts.

Next, the scoring phase ranks nodes based on your affinity rules. Proper rules ensure pods do not crowd on a single host. You can also use topology spread constraints to distribute pods evenly across zones.

If you face heavy resource pressure, you must protect your workloads. Learn about kubernetes scheduling priority and preemption to keep your key apps online. Good rules stop node overload before it starts.

When a Managed Control Plane Beats Hand-Rolled Scheduling

As you run more apps on your host nodes, kubernetes scheduling gets hard to track. Writing manual rules for every single node leads to mistakes, and soon your team spends all their time trying to fix pods. Doing this by hand slows you down. That is why using a managed control plane makes so much sense for growing teams.

The Pain of Scale

At first, small teams can manage a few scheduling tasks on their own. But when you grow to ten or more clusters, the work gets too big. Each cluster needs its own taints, labels, and rules, making it hard to keep things the same. Node setups can go wrong. If you do this by hand, nodes can sit empty while some pods starve. In fact, a paper from the University of Washington shows that bad node setup hurt speed in multi-cluster systems. Hand-rolled rules often break. Delegating these tasks to a smart control plane helps you scale without the headache.

The Pull Model for Regulated Fleets

For fields like finance, healthcare, and government, safety is the main goal. These firms must follow strict laws. They need to run their fleets in secure, air-gapped areas where they can guard their data. Standard scheduling tools fail here because they need a direct path to the cloud. Plural solves this with a self-hosted, agent-based pull design. It runs behind your firewalls and only pulls data out, so no secret keys leave your site. This matches guidelines like those in the NIST container security guide, which warns of risk from central tools. With this egress-only model, you get full control and peace of mind.

A Unified Control Plane

Instead of fighting with YAML files, you can manage your whole fleet from a single pane of glass. Plural is an AI-native platform built to run at scale. It combines Kubernetes CD with IaC tools like Terraform, Pulumi, and Ansible, and adds AI to run tasks for you. Check out our simple pricing model. You can also read our technical blog for deep guides. By using a managed control plane, your team does not have to worry about the fine points of scheduling. This lets them focus on writing code that helps your business grow.

Ready to simplify your Kubernetes fleet? Contact our team today to schedule a free demo.

Talk to a Plural engineer about bringing your kubernetes scheduling and fleet management onto a managed control plane.

Frequently Asked Questions

How does the kube-scheduler select the best node for a pod?

The selection process runs in two main steps: filtering and scoring. In the first step, the scheduler filters out nodes that do not meet pod requirements. In the second step, it ranks the remaining nodes to find the best fit. According to the authoritative Kubernetes documentation, these steps are known as predicates and priorities.

How do taints and tolerations affect kubernetes scheduling?

Taints are applied to nodes to repel pods that do not have a matching toleration. Tolerations are declared on pods, allowing them to schedule on nodes with matching taints. This setup prevents general workloads from running on dedicated nodes. You can learn more about kubernetes scheduling with tolerations to optimize your resource usage.

How does pod priority affect kubernetes scheduling?

When a cluster is full, the scheduler can evict low-priority pods to make room for high-priority ones. This process of eviction is called preemption. It helps ensure that critical workloads get the CPU and memory they need. You can learn more about kubernetes scheduling priority and preemption to manage your resources.

How does the cluster autoscaler interact with kubernetes scheduling?

The scheduler maps pods to nodes that have enough resources. If no node has enough space, the pods remain in a pending state. The autoscaler watches for these pending pods and spins up new nodes to host them. You can read about kubernetes scheduling and autoscaling to learn how they work together to prevent downtime.

Ready to simplify Kubernetes scheduling?

Setting up scheduling rules by hand across ten or more production clusters takes too much time and slows down your daily business growth. Small mistakes in your custom YAML configuration files can easily cause major cluster downtime or waste your cloud budget on idle nodes. You can automate this heavy burden today to secure your systems, lower your cloud bills, and free up your engineers.

Ready to automate? Please contact Plural today to schedule a quick conversation. Talk to a Plural engineer today about delegating your complex Kubernetes scheduling and fleet management tasks to our managed control plane. Our self-hosted platform is built to handle massive fleets with no central credential storage for complete peace of mind.