Kubernetes Exit Code 137: Causes, Fixes, and Prevention

A container that exits with status 137 has not returned a normal application error. It was terminated by signal 9, SIGKILL, which stops the process immediately and cannot be caught or handled. The number follows the Unix convention of 128 plus the fatal signal number, so 128 + 9 produces 137. In Kubernetes, the most common explanation is that the container exceeded its memory limit and was marked OOMKilled, but the status alone is not a complete diagnosis.

That distinction matters in production fleets: a cgroup limit, an evicted pod, and a host under severe pressure require different fixes. Start by separating the signal's meaning from the mechanism that delivered it, then trace the failure through Kubernetes status and resource data.

Start a free demo of Plural to see memory and OOMKilled risk across your Kubernetes fleets.

What Is Kubernetes Exit Code 137, and Why Does It Happen?

Kubernetes exit code 137 means the container's main process was terminated with SIGKILL, Linux signal 9. The number is not an application-specific error code. It is the shell's conventional representation of a process that ended because it received a fatal signal: 128 plus the signal number. In this case, 128 + 9 = 137. The NASA Advanced Supercomputing guidance describes the same calculation, and the SDSU Bash reference documents the broader 128+n convention.

SIGKILL is a hard stop. Unlike SIGTERM, it cannot be caught, ignored, or handled by the application. Kubernetes cannot give the process a graceful shutdown period after the kernel has issued this signal. Any in-memory state that the process has not persisted is lost, and the container runtime reports the resulting status to Kubernetes as exit code 137.

How does a memory limit produce exit code 137?

The most common path is a container exceeding its cgroup memory limit. Kubernetes uses Linux control groups to enforce the memory limits defined for containers. When a process inside the cgroup consumes more memory than that limit allows, the kernel's out-of-memory mechanism terminates a process to protect the node. Kubernetes then commonly exposes the termination reason as OOMKilled, with exit code 137 recorded for the container.

For example, a service may have a 512 MiB memory limit while its runtime, heap, buffers, and working set grow beyond that boundary during a traffic spike. The application may still be responding normally immediately before termination. The kernel does not wait for the service to return an error or for Kubernetes to resize the container. It enforces the cgroup boundary, sends SIGKILL, and the process exits with status 137.

This distinction matters when investigating the failure. The number tells you how the process ended, not the complete reason it was killed. An OOMKilled reason in the pod's last state strongly connects the signal to a memory-limit violation. Without that status, exit code 137 only establishes that SIGKILL was involved. Other system-level actions can also send SIGKILL. So operators should verify the pod status and event history rather than treating the number alone as proof of an out-of-memory event.

What should you check first?

Start with the terminated container's last state and the pod events. Commands such as kubectl describe pod and kubectl get events can show whether Kubernetes recorded OOMKilled, a memory-pressure event, or another termination path. Then compare the container's observed memory usage with its configured request and limit. That evidence separates a process that needs a higher limit from one with a leak, an unexpectedly large workload, or an external kill condition.

Common Causes of Kubernetes Exit Code 137

Exit code 137 means the process received SIGKILL, signal 9. In Kubernetes, the most common path is an out-of-memory condition: the Linux kernel terminates a process when it exceeds the memory boundary enforced for its container. SIGKILL cannot be caught or handled by the application, so the process stops immediately. The numeric relationship is 137 = 128 + 9, where 9 identifies the fatal signal. See the signal and exit-code explanation from NASA for the underlying model.

That common pattern has several operational causes. The error is not a diagnosis by itself. It is the final symptom of a process being forcibly terminated, and the source of the memory pressure determines which remediation is appropriate.

The container exceeded its cgroup memory limit

Each container runs inside a Linux control group, or cgroup, with resource boundaries established by the container runtime and Kubernetes configuration. When the process attempts to use more memory than its cgroup limit allows, the kernel enforces the boundary. It can issue SIGKILL to the process, and Kubernetes then reports a terminated container with the reason OOMKilled and an exit code of 137.

This is the answer to the common question. "What happens when a container hits a cgroup memory violation?" The kernel does not wait for the application to release memory or finish a graceful shutdown. It protects the cgroup boundary by killing the process. A memory leak, an unexpectedly large workload, an undersized Java heap, or a burst in concurrency can all push a container past its configured limit. The relevant limit belongs to the container, not necessarily to the entire node.

The kubelet evicted the pod under node memory pressure

A pod can also be terminated because the node is under sustained memory pressure. The kubelet monitors node conditions and can set MemoryPressure when available memory falls below eviction thresholds. To protect node stability, it evicts pods and asks their containers to stop. Depending on the termination path and what process is observed, the resulting workload symptoms can resemble an OOM kill, including a 137 status.

Eviction is different from a container simply crossing its own limit. The pod may have been within its declared limit while the node as a whole became constrained. Low requests can allow too many workloads to be scheduled together, while system daemons, image layers, or another workload consume the remaining capacity. Check node conditions and pod events before changing only the application limit.

The host node ran out of memory

At the broadest level, the node itself may exhaust memory. The host-level OOM killer can select a process when the operating system cannot satisfy allocation requests. This can affect a container, a node service, or another process, and it may occur even when the application has not obviously exceeded its intended workload profile. Overcommitted capacity, missing limits, and memory-heavy sidecars increase the risk.

Fleet-wide Kubernetes observability helps separate container-level growth from node-level pressure by correlating working-set metrics, pod events, and node conditions. For restricted environments, an agent-based Kubernetes security model can also support in-cluster visibility without requiring inbound access. The next step is to verify which termination path occurred before changing resource settings.

How to Diagnose Exit Code 137: Events, Status, and Metrics

Once a container returns exit code 137, confirm what terminated it before changing resource limits or restarting workloads. The code indicates termination by signal 9, or SIGKILL, under the conventional 128+n exit-code model. In Kubernetes, the most common explanation is an out-of-memory kill. But the pod status and node evidence should establish whether the container exceeded its limit or the node was under broader memory pressure. The following sequence keeps the investigation tied to the affected pod and its runtime context.

1. Check the pod and container status. Start with the namespace and pod name, then inspect the current state and restart count:

   kubectl get pod <pod-name> -n <namespace> -o wide
   kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState}'

   Look for reason: OOMKilled in the terminated or last state fields. That status is the clearest confirmation that the container was terminated after exceeding its memory limit. Record the restart count, node name, and timestamps. A recent deployment, traffic increase, or batch job can explain why the failure began when it did.

2. Describe the pod and inspect its event history. Run kubectl describe pod to see container state, exit code, resource configuration, scheduling details, and events in one view. The inspecting pod status events guide provides a more detailed walkthrough of this command. Compare the container's configured memory limit with its recent behavior, and check whether the pod was rescheduled or repeatedly restarted. Pay particular attention to the Last State block, termination reason, and termination time.

   kubectl describe pod <pod-name> -n <namespace>

3. Query Kubernetes events across the relevant scope. Events can distinguish a container-level OOM kill from node pressure, eviction, failed probes, or another lifecycle event. Sort them by creation time and filter for the namespace or involved object:

   kubectl get events -n <namespace> --sort-by=.lastTimestamp
   kubectl get events --all-namespaces --sort-by=.lastTimestamp | grep -iE 'oom|evict|memory|kill'

   Use the investigating OOM killed pods guide when you need to correlate warnings with the pod's restart timeline. If the event points to node memory pressure or eviction, inspect the node as well rather than treating the workload limit as the only cause.

4. Pull memory metrics for the container. Check time-series data for container_memory_working_set_bytes around the termination timestamp. Compare the working set with the container's memory limit and look for a steady climb, a short spike, or pressure affecting multiple workloads. A single-container rise toward its limit suggests application demand or a leak. Simultaneous rises across pods suggest node-level contention or insufficient capacity. If metrics are unavailable, fix that observability gap before relying on restart counts alone.
5. Check kubelet and application logs. Review kubelet logs on the affected node for memory-pressure, eviction, or cgroup messages. Application logs from the previous container can reveal the request or workload phase immediately before termination. Use troubleshooting pod failures to retrieve logs from the terminated instance, for example with kubectl logs <pod-name> -n <namespace> --previous. Correlate those logs with metrics and events before deciding whether to tune the limit, reduce concurrency, or address node capacity.

This evidence-based sequence turns kubernetes exit code 137 from a restart symptom into a specific diagnosis: an application limit breach, node memory pressure, or another SIGKILL source.

Is Exit Code 137 Always Caused by OOMKilled?

No. OOMKilled is the most common explanation for kubernetes exit code 137, but the status alone only tells you that the container's main process ended with signal 9, SIGKILL. The conventional exit-status calculation is 128 plus the signal number, so 128 + 9 produces 137. It does not identify which actor sent the signal.

That distinction matters when an application is restarting in production. Treating every 137 as a container memory-limit violation can lead you to increase limits while the real issue is an operational action. Node instability, or a different layer of resource pressure.

When OOMKilled is the likely cause

If the container exceeded its cgroup memory limit, the Linux kernel can terminate it with SIGKILL. Kubernetes normally exposes that cause in the terminated container state as reason: OOMKilled. A matching event, rising memory metric, or repeatable failure near the configured limit strengthens the diagnosis. In this case, inspect the workload's memory request and limit, then compare them with the process's working set and peak usage before changing the limit.

Other ways a process can end with 137

A direct administrative action can produce the same signal. For example, an operator or automation may send kill -9, or a deletion workflow may terminate the container abruptly while a rollout, node drain, or emergency remediation is in progress. A node reboot or sudden host failure can also interrupt a process before it performs a graceful shutdown.

Node-wide pressure adds another layer of nuance. A sibling pod can consume memory on the same node. Contributing to host-level pressure and an OOM decision even when the affected container has not obviously crossed its own limit. Kubelet eviction and node-level memory management can therefore make the node's condition as important as the pod's specification. Review node conditions, eviction events, and neighboring workloads rather than looking only at the failing deployment.

Finally, distinguish a real SIGKILL from an application or wrapper that deliberately reports status 137 after handling another signal. SIGKILL cannot be caught or handled by the process itself. A shell script, supervisor, or application can nevertheless exit with a chosen numeric status, which can make the record look like a kernel kill without proving one occurred.

Use the termination reason, not the number alone

Start with kubectl describe pod and the container's last termination state. If Kubernetes reports OOMKilled, investigate memory accounting. If it reports a different reason, correlate pod events with node conditions, rollout history, deletion activity, and deployment automation. This is also where a fleet-level view helps. Plural's Kubernetes management platform can connect workload operations and cluster observability across environments, so teams can compare an isolated OOM event with a broader node or rollout incident.

The practical rule is simple: use 137 as a clue that SIGKILL was involved, then use Kubernetes status, events, and node telemetry to establish why.

How to Prevent Kubernetes Exit Code 137 with Resource Requests and Limits

Prevention starts with making memory behavior explicit in the Pod specification. A memory request tells the Kubernetes scheduler how much memory a container needs when it places the Pod. A memory limit establishes the container's upper boundary. If the container exceeds that limit, Kubernetes can terminate it with an out-of-memory kill, producing the SIGKILL-derived exit code 137. The NASA High End Computing Capability documentation describes 137 as 128 plus signal 9, or SIGKILL: https://www.nas.nasa.gov/hecc/support/kb/print/185/.

Requests and limits solve different problems, so setting only one creates an incomplete policy. A request that is too low can place a memory-intensive workload on a node without enough practical headroom. A limit that is too low can kill a healthy application during a predictable traffic spike. Conversely, a Pod without a meaningful limit can consume memory until the node is under pressure. Putting other workloads at risk and making the eventual failure harder to attribute.

Use requests for placement and limits for the OOM threshold

Start with measured usage, then leave room for normal variance and known peaks. The following Deployment gives the scheduler a 256 MiB placement signal and sets 512 MiB as the container's memory ceiling:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: example/api:1.0
          resources:
            requests:
              memory: "256Mi"
            limits:
              memory: "512Mi"

The request does not reserve an exact, permanent amount of RAM inside the application. It is a scheduling input used to select a node with sufficient allocatable capacity. The limit is enforced through the container's control group. Once usage crosses the configured memory limit, the kernel and Kubernetes runtime can kill the process. That distinction matters during incident review: a Pod can be scheduled successfully because its request fits, then later exit 137 because its actual working set exceeds its limit.

Do not copy these values blindly. Compare the request and limit with observed working-set memory, startup behavior, cache growth, and concurrency. If a service regularly approaches its limit, first determine whether the application has a leak, an unbounded cache, or a legitimate capacity requirement. Raising the limit may prevent immediate restarts, but it does not fix a leak or ensure the node has capacity to sustain the workload. Revisit requests as usage changes so scheduling decisions continue to reflect production reality.

At fleet scale, resource policy is easier to maintain when it is visible in version control and reviewed alongside application changes. Pair per-workload manifests with cluster-wide observability and cost analysis, such as this guide to right-sizing Kubernetes cost. Plural also provides a unified operational view and documentation for managing Kubernetes environments. If repeated exit 137 incidents span multiple clusters, review your fleet with Plural and identify where requests, limits, and actual consumption have diverged.

Best Practices for Production Memory Management

Production memory management is a capacity-planning problem, not a restart policy. A container that reaches its memory limit can be terminated with SIGKILL and surface as Kubernetes exit code 137. The durable fix is to make memory behavior explicit, measure it continuously, and apply the same operating model across every cluster.

Set a request and limit for every container

Start with a resource contract in each workload specification. The request tells the scheduler how much memory the container needs for placement. The limit defines the maximum it may consume before the runtime and kernel can terminate it. Leaving either field blank makes it harder to reason about scheduling, eviction, and failure behavior.

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Choose values from observed working-set usage rather than copying a default across deployments. A request that is too low creates unstable placement and increases contention. A limit that is too tight creates avoidable OOMKilled restarts. A limit that is too high can allow one workload to crowd out others on a node.

Leave headroom at the node level

Pod limits are not the same as node capacity. Nodes reserve memory for the operating system, kubelet, container runtime, and DaemonSets. Plan against allocatable memory, not the headline capacity shown by the instance type. Keep enough headroom for short-lived spikes, rolling deployments, and system processes. Otherwise, a cluster can look adequately provisioned in aggregate while individual nodes enter memory pressure and begin evicting pods.

Review memory requests and limits after meaningful workload changes, not only after an incident. Track container working-set usage, restart counts, eviction events, and the gap between requested and consumed memory. Working set is more useful for operational decisions than a single instantaneous usage sample because it reflects memory the container is actively using and cannot readily reclaim.

Right-size before adding restart logic

Repeatedly restarting a container that has a stable memory-growth pattern only creates an OOM loop. Use historical metrics to distinguish a steady leak from a bursty batch workload, then adjust the request, limit, application configuration, or code path. Vertical Pod Autoscaling can recommend or apply resource values when its policy fits the workload. For services with predictable traffic, explicit right-sizing and load tests may be safer. Treat every automated recommendation as an input for review, especially for latency-sensitive or regulated workloads.

Fleet-wide visibility matters once the same service runs across many clusters. A central view should let operators compare working-set trends, limits, restarts, and node pressure without switching between dashboards. Plural's unified control plane provides a single pane of glass for that operating model, while its Kubernetes observability guidance shows how to connect metrics with actionable investigation.

See how Plural helps platform teams manage memory risk across Kubernetes fleets.

Finally, include memory checks in deployment and upgrade workflows. Validate resource changes before rollout, watch pressure during the rollout window, and preserve rollback paths. Pair that discipline with Kubernetes upgrading practices so infrastructure changes do not turn a hidden capacity mismatch into a fleet-wide restart event.

Frequently Asked Questions

What does exit code 137 mean in Kubernetes?

It means the container's main process was terminated with SIGKILL, signal 9. The 137 value follows the 128+n convention, so 137 equals 128 plus 9. In Kubernetes, the most common explanation is OOMKilled: the container exceeded its memory limit and the kernel terminated it. NASA documents the signal-to-exit-code relationship.

How do you fix Kubernetes exit code 137?

First confirm the cause instead of raising the limit immediately. Check the pod's termination reason, events, logs, and memory metrics. If the container exceeded a valid limit, profile the application, reduce its peak usage, or increase the limit to a value the node can support. Then set a realistic memory request so the scheduler places the workload on a suitable node.

Is exit code 137 always caused by OOMKilled?

No. Exit code 137 identifies SIGKILL, not the specific actor that sent it. OOMKilled is the usual Kubernetes cause, but a process can also be killed by another system action or by resource pressure outside the container's own limit. Use the pod status and event history to distinguish a cgroup memory kill from eviction, node-level pressure, or an explicit termination.

Does a Kubernetes memory request affect exit code 137?

Yes, but indirectly. A request helps Kubernetes schedule the pod onto a node with the required allocatable memory. The memory limit establishes the container's enforced ceiling, so exceeding it can trigger OOMKilled and exit code 137. Configure both from measured usage, and leave headroom for normal peaks rather than relying on an unlimited or arbitrary value.

Ready to manage Kubernetes memory issues across your fleet?

Exit code 137 is easier to prevent when teams can see memory pressure, compare workloads, and apply consistent resource policies across clusters. Get a hands-on demo of Plural's Kubernetes fleet management platform to unify observability and automate resource right-sizing. Get started with Plural to see how a unified control plane can support more reliable day-2 operations.