Kubernetes OOMKilled: How to Diagnose and Fix It

A pod that restarts without an application error can turn a routine deployment into a debugging exercise. When Kubernetes reports a container as OOMKilled, the immediate question is whether the container exceeded its cgroup memory limit or whether the node itself ran out of memory. Those cases look similar in a dashboard, but they require different fixes.

The reliable path is to establish what was killed, identify which memory boundary was crossed, and then adjust limits, requests, or the application itself. Start with the termination status and exit code, then trace how Kubernetes and Linux account for memory under pressure.

Explore how Plural helps you diagnose and fix Kubernetes OOMKilled issues and related memory pressure.

What Is Kubernetes OOMKilled and What Does Exit Code 137 Mean?

OOMKilled is Kubernetes reporting that a container was terminated after running out of memory. More precisely, the Linux kernel's Out-Of-Memory (OOM) killer stopped the container process because it exceeded the memory available to its control group, commonly the container's configured memory limit. Kubernetes then records the termination reason as OOMKilled and exposes exit code 137 for the terminated process. The underlying distinction matters: Kubernetes surfaces the event, but the Linux kernel performs the kill.

Exit code 137 is the numeric result of a process receiving signal 9, or SIGKILL. In container environments, it is a standard indicator that the process was killed externally, typically because the system was under memory pressure. That makes 137 useful evidence, but not a complete diagnosis. The code tells you how the process ended. It does not, by itself, tell you whether the container hit its own cgroup limit or whether the node was already out of memory.

For the first case, imagine a container with a memory limit of 512Mi. If the application grows beyond that limit, the kernel can terminate it even when the Kubernetes node still has unused memory. The container then restarts according to its pod policy, and a deployment may appear healthy at a glance while one or more replicas repeatedly cycle. Transient allocation spikes, unbounded caches, and memory leaks can all produce this pattern. A limit that is too close to normal peak usage leaves little room for legitimate variation.

The second case is node-level pressure. A node's physical memory is shared by the workloads scheduled there, so several containers can contend for the same finite resource. When the entire node is exhausted, the kernel may kill a process even when that process has not crossed its individual limit. This is why an investigation should examine both the container configuration and node memory conditions rather than treating every 137 as a bad application limit.

Memory exhaustion is therefore a Linux resource-management event expressed through Kubernetes state. The kernel's OOM killer exists to protect system stability by terminating processes when memory cannot be allocated safely. Kubernetes resource settings influence when that boundary is reached, but they do not replace host-level memory management. For a fleet-wide view, Plural provides a unified Kubernetes control plane that can help platform teams connect workload state with cluster operations. Pair that view with focused Kubernetes observability and metrics practices before deciding whether to raise a limit, right-size a request, or address a broader node-capacity problem.

Why Does the Linux OOM Killer Terminate a Container?

A Kubernetes OOMKilled event is usually the visible result of a memory decision made by Linux. Kubernetes applies the memory policy declared for a container through the Linux control group, or cgroup. When the process uses more memory than its cgroup limit allows, the kernel terminates a process in that cgroup. The container then exits, commonly with status code 137, and Kubernetes may restart it according to the workload's restart policy.

This is why increasing a deployment's replica count or restarting the pod does not necessarily solve the problem. If the application repeatedly exceeds the same boundary, each replacement pod encounters the same limit. The limit is a protection mechanism, but a limit set below the application's real working set converts normal allocation spikes into restarts. The relationship between a container's configured memory limit and how the Linux kernel enforces it is what produces the OOMKilled status and exit code 137.

Requests and limits serve different purposes. A memory request is the amount Kubernetes uses when scheduling the pod and the amount of memory the container is guaranteed to receive under normal resource accounting. A memory limit is the maximum memory the container may consume. For example:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

With this configuration, the scheduler evaluates the pod against a 512 MiB request, while the container is capped at 1 GiB. A request is not a reservation of unlimited burst capacity. And a limit does not guarantee that the node has that amount of physical memory available at every moment. Treating the two fields as interchangeable often produces either poor bin-packing or unexpected OOMKilled restarts.

When the node runs out of memory

The cgroup limit is only one boundary. A node's physical memory is shared by every pod on that node, plus the kubelet, the container runtime, operating-system processes, and DaemonSets. Pods therefore contend for the same finite resource. If total node demand exceeds available memory, the kernel can invoke the system-level OOM killer even when the affected container has not crossed its own configured limit. In that case, the event is a node-capacity problem rather than proof that the container's limit alone was too small.

For diagnosis, compare the container's configured limit with node-level memory usage and allocatable capacity. Querying node resource usage helps distinguish a workload-specific cgroup breach from cluster-wide contention. A fleet view through Plural's Kubernetes management console can make the same comparison practical across multiple clusters instead of requiring isolated checks on each node.

Requests also affect what happens during node memory pressure. Pods without meaningful memory requests are more vulnerable to eviction because Kubernetes has less scheduling information and they receive a less favorable position in resource-pressure decisions. Define requests from observed baseline usage, then set limits high enough to accommodate legitimate peaks. This does not eliminate kernel OOM events, but it gives the scheduler better placement data and reduces avoidable contention.

How to Confirm a Kubernetes OOMKilled Pod With kubectl

Before changing a deployment, confirm that the restart was caused by memory exhaustion rather than an application crash, probe failure, or manual termination. Kubernetes exposes the evidence at the container, pod, event, and node levels. Reading those signals together tells you whether the container crossed its cgroup limit or the node itself was under memory pressure.

1. Inspect the container state

   Start with the pod description in the namespace where the workload runs:

   kubectl describe pod POD_NAME -n NAMESPACE

   In the Containers section, inspect State, Last State, Reason, Exit Code, and restart counts. A previous container state with Reason: OOMKilled confirms that the container was terminated for exceeding its available memory. Exit code 137 is consistent with an external termination caused by memory pressure, although the reason field is the more direct Kubernetes signal. This is the first check for confirming an OOMKilled container and for correlating the reason with the exit code.

2. Check namespace events

   Events provide timing and context around the failure:

   kubectl get events -n NAMESPACE --sort-by=.lastTimestamp

   Look for entries associated with the pod or its workload that report an OOMKilled reason, memory pressure, eviction, or a related node condition. Compare the event timestamp with the container restart time from kubectl describe. If the event stream points to node pressure or eviction instead of a container limit breach, investigate the node before editing the pod specification.

3. Read the pod YAML for the complete record

   Use the API representation to inspect status fields and resource configuration:

   kubectl get pod POD_NAME -n NAMESPACE -o yaml

   Under status.containerStatuses, check lastState.terminated.reason, exitCode, finishedAt, and restartCount. Then compare those fields with each container's resources.requests.memory and resources.limits.memory in the pod specification. A limit caps the memory a container can consume, while a request describes its guaranteed allocation. Checking both the observed termination and declared resources prevents a diagnosis based on status alone.

4. Separate container pressure from node pressure

   Finally, check memory at the node level:

   kubectl top pod POD_NAME -n NAMESPACE
kubectl top node NODE_NAME

   These commands require the Metrics Server. Use the Kubernetes Metrics Server guide to inspect pod memory usage and the kubectl top nodes guide to query node resource usage. Compare the node's available capacity and other pods' consumption with the failing container's limit. Effective OOMKilled troubleshooting requires both container limits and node memory metrics. Because pods share the node's physical memory and a node-wide shortage can produce kernel-level kills outside the pod's individual limit.

Once these checks agree, you have enough evidence to choose the next action: right-size the workload's memory request and limit. Investigate application growth, or remediate node capacity and scheduling pressure.

How to Fix Kubernetes OOMKilled Errors

Once you have confirmed that a container was terminated for exceeding its available memory, fix the constraint instead of treating the restart as an isolated incident. The right solution depends on whether the container limit is too low. The request does not reflect normal usage, or the application is consuming more memory over time than expected.

Start by reviewing the workload's recent memory profile and its resource configuration. A useful baseline is the amount of memory the container needs during ordinary operation, including expected concurrency and routine background work. Set the memory request close to that baseline so Kubernetes can schedule the workload on a node with appropriate capacity. Then set a limit that leaves room for legitimate variation. Requests provide a guaranteed allocation, while limits cap maximum consumption. Both values need to reflect how the application actually behaves, not an arbitrary default.

For example, a deployment might begin with a deliberately conservative configuration:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

The values are only a starting point. Measure the workload before choosing production settings, and revise them as traffic and application behavior change. A limit that is too strict can turn a short-lived memory spike into an OOMKilled restart. Raising the limit may stop those transient failures, but it does not solve a leak or an unexpectedly growing working set. It can also move the failure to the node if several pods collectively consume more memory than the node can provide.

Right-sizing requests is therefore a scheduling and stability decision, not just a way to silence alerts. If requests are set below the application's baseline, Kubernetes may place workloads on nodes without enough practical headroom. Pods without properly defined memory requests are also more exposed when a node enters memory pressure. Compare container-level usage with node-level capacity before deciding whether to increase one pod's limit, redistribute workloads, or add capacity.

Next, investigate the application itself. Reduce unnecessary in-memory caching, bound queues and batch sizes, and review code paths that retain objects longer than intended. Compare memory growth across repeated deployments and representative workloads. A steadily rising working set points toward a leak or retention problem, while short, repeatable peaks may indicate a need for better buffering or a less restrictive limit. Monitoring memory growth over time helps teams find leaks before they trigger an OOMKilled event.

At fleet scale, manual tuning does not remain reliable for long. Plural can provide a unified view of workload behavior and support right-sizing decisions across clusters. While its observability capabilities help platform teams connect pod failures to broader resource pressure. Review Plural's pricing and capacity automation options when you need a more consistent approach to Kubernetes day-2 operations.

See Plural's pricing and capacity options for automating Kubernetes memory right-sizing.

Requests vs. Limits: Right-Sizing Kubernetes Memory

Memory requests and memory limits solve different problems, and confusing them can make a workload look healthy until the node is under pressure. A request tells the Kubernetes scheduler how much memory a container needs as a baseline. A limit sets the maximum memory the container may consume. Together, they define placement and runtime behavior, but neither value should be copied from a generic example.

The scheduler reasons about requests, not the memory limit alone. If a deployment specifies a 256Mi request and a 1Gi limit, Kubernetes schedules the pod as a 256Mi workload even though the process may grow to 1Gi. That can be efficient when the baseline is accurate, but it can also create contention if many workloads routinely expand beyond their requests.

Missing or negligible requests create a different risk. During node memory pressure, pods without meaningful requests are more exposed to eviction because Kubernetes has less information about the capacity they require and their relative priority. A request is therefore not just a placement hint. It is part of the stability contract between the workload and the scheduler.

Right-size the request to observed baseline memory rather than to the smallest successful test run. Capture normal usage across representative traffic, startup, cache warm-up, and scheduled jobs. Then set the request near the workload's sustained baseline, leaving the limit high enough for legitimate short-lived growth. This approach gives the scheduler a useful capacity signal without turning every transient spike into a restart. The principle is supported by the distinction between Kubernetes Resource Metrics and custom metrics used for capacity planning and autoscaling in Kubernetes resource management research.

Do not treat a larger limit as a fix for every kubernetes oomkilled event. If the baseline is rising, investigate application allocation patterns, leaks, caches, and workload concurrency. If the node itself is exhausted, increasing one pod's limit can intensify contention rather than solve it. Compare container usage with node capacity, then revise requests and limits together so scheduling and runtime enforcement describe the same workload.

How to Monitor Memory Pressure and Prevent OOMKilled

Preventing OOMKilled incidents requires more than checking whether a container has a memory limit. You need visibility at three levels: the node, the pod, and the application. A pod can remain within its configured limit while the node approaches total memory exhaustion. Or an application can show a steady upward trend that is invisible in a single point-in-time check. Monitoring must expose all three conditions early enough for an operator or autoscaler to respond.

Start with Kubernetes Resource Metrics. These default metrics include CPU and memory usage for host machines and pods, giving you the baseline needed for capacity planning and workload monitoring. The Kubernetes metrics server supplies this data to the Horizontal Pod Autoscaler (HPA), which can adjust replica counts as demand changes. HPA is not a substitute for correctly sized limits, but adding replicas can prevent a traffic spike from concentrating demand in a single pod and exhausting its memory. Research on Kubernetes autoscaling distinguishes Resource Metrics from custom metrics and highlights their role in HPA behavior and performance (see the analysis of Kubernetes resource metrics and HPA).

Alert on trends, not only failures

An alert that fires after a pod has already been OOMKilled is an incident notification, not prevention. Alert on sustained pod memory utilization, rapid memory growth, and node-level available memory. A workload that climbs steadily after every deployment may indicate a leak. A workload that repeatedly approaches its limit during normal peaks may need a higher limit, application optimization, or more replicas. Kubernetes guidance treats advanced monitoring and alerting as a way to identify memory trends before they cross the thresholds associated with OOMKilled events.

Use Prometheus when the default resource view is too coarse. Custom metrics can represent application-specific demand, such as queue depth, request concurrency, or cache size, and can give autoscaling finer control than CPU or memory utilization alone. Prometheus-backed alerts should distinguish a pod nearing its limit from a node under broad memory pressure. That distinction matters because the remediation differs: one may require right-sizing or debugging, while the other may require rescheduling workloads or adding node capacity. The difference between Kubernetes Resource Metrics and Prometheus Custom Metrics is documented in the same academic analysis (resource and custom metrics comparison).

Make the signal operational across the fleet

For teams operating multiple clusters, centralizing these signals reduces the time between detection and action. Plural provides fleet-wide observability through a single-pane-of-glass console, so operators can compare memory pressure across clusters instead of switching between isolated dashboards. Pair alerts with documented runbooks: inspect recent deployments, compare pod and node memory, review replica behavior, and verify that requests and limits reflect observed usage. For broader capacity planning and resource automation, see Plural's platform engineering resources.

Finally, test the alert path before an outage. Confirm that alerts identify the affected namespace, workload, pod, cluster, and current memory condition, then route them to the team responsible for remediation. The goal is not to eliminate every memory spike. It is to detect pressure while you still have options: scale horizontally, adjust capacity, correct a leak, or revise resource settings before Kubernetes has to terminate the container.

Discover Plural's fleet-wide observability for monitoring Kubernetes memory pressure across clusters.

Frequently Asked Questions

What is an OOMKilled error in Kubernetes?

OOMKilled means the Linux kernel terminated a container after it exceeded the memory available to it, commonly its configured memory limit. Kubernetes then reports the termination in the container state, and the workload may restart according to its restart policy.

What does exit code 137 mean in Kubernetes?

Exit code 137 means the container process was killed externally, typically by the kernel during memory pressure. Treat it as a signal to investigate both the container's configured limit and the node's overall memory usage, rather than assuming the application exited normally.

How do I troubleshoot OOMKilled pods?

Start with kubectl describe pod POD_NAME -n NAMESPACE and inspect the container state and Events section for OOMKilled. Then compare the pod's memory request and limit with current container and node metrics. If the node is under pressure, the cause may be node-wide contention rather than a single container limit.

How can I prevent Kubernetes OOMKilled errors?

Set a memory request that reflects the workload's baseline usage and a limit that accommodates legitimate peaks. Measure memory growth over time, investigate leaks, optimize the application, and alert on sustained memory pressure before the kernel must terminate a process.

What is the difference between Kubernetes limits and requests?

A memory request is the amount Kubernetes uses when scheduling a pod and treats as the container's guaranteed allocation. A memory limit caps how much memory the container can consume. Requests support placement decisions, while limits constrain runtime usage.

Ready to manage Kubernetes memory across your fleet?

Persistent OOMKilled errors are easier to address when platform teams can connect workload behavior, resource settings, and cluster health in one operational view. Explore Plural as your unified control plane to get started with a more consistent way to manage Kubernetes day-2 operations across your fleet.