Kubernetes 502 Bad Gateway: a command-driven path from Ingress to EndpointSlice

A 502 that starts immediately after a deployment, appears only on one route, or spikes under load is almost never a browser problem. It is a proxy telling you that it selected an upstream and could not get a usable HTTP response back. In Kubernetes, that proxy may be an Ingress controller, a gateway, or a service-mesh sidecar; the upstream may be a Service endpoint, a pod, or another proxy.

When I work a 502, I do not restart the Deployment first. A restart can temporarily change the symptom and erase the evidence I need. I start with the proxy that emitted the response, then follow one request toward the application: Ingress rule, Service, EndpointSlice, pod readiness, listening socket, and application or sidecar logs. That sequence makes the distinction between a routing failure and an application failure visible.

A Running pod is not proof of a healthy upstream. It may be unready, listening on the wrong port, bound only to 127.0.0.1, closing connections during shutdown, or returning malformed HTTP. A process that answers a request from inside its own container can still be unreachable through a Service if it does not bind to an address such as 0.0.0.0. The Kubernetes Services documentation is the reference for how Services select and route to pods.

What a 502 tells you in Kubernetes

A 502 identifies the boundary at which a gateway received an unusable upstream response. It does not establish that the application is down, and it does not necessarily mean the Ingress controller is broken. The same client-facing status can come from a refused connection, an upstream timeout, a connection reset while a pod is terminating, or incomplete HTTP from the backend.

• Connection refused: the process crashed, never started listening, binds to the wrong address, or the Service sends traffic to the wrong port.
• Timeout: the backend is overloaded, blocked on a dependency, saturated on threads or database connections, or legitimately exceeds the proxy timeout.
• Premature close: the backend closes the connection before it returns a response, often during a rollout or a process crash.
• Invalid response: the backend returns incomplete or malformed HTTP that the proxy cannot parse.

During a rollout, endpoint propagation adds one more wrinkle. A terminating pod can remain briefly in an EndpointSlice or in the Ingress controller's routing view after it stops accepting new work. Kubernetes documents the associated endpoint behavior in its Service networking documentation. That explains a short burst of intermittent 502s after a deploy; it does not make the burst harmless.

Common 502 failure modes

I use the symptom to prioritize the next check, but I still verify the request path. A failure on every host and path needs a different investigation from a failure on one Service after a single Deployment changed.

Trace the request path before changing configuration

The route is not abstract: client to Ingress controller, Ingress controller to Service, Service to EndpointSlice, EndpointSlice to pod. In a mesh, there is another proxy boundary on one or both sides. I write down the failing host, path, namespace, Service, and a UTC timestamp before I change anything. Without that timestamp, it is easy to compare the wrong log line to the wrong rollout.

Client to Ingress controller

The external load balancer or Ingress controller matches the request host and path against the configured rules, then proxies it upstream. A wrong host rule, a conflicting path, an unavailable controller, or a backend-port mismatch can stop the request at this first hop. If only one route fails, start with that Ingress object and the controller logs rather than the Deployment. This guide to troubleshooting ingress controllers is useful when you need to account for the fields and events in kubectl describe ingress.

Ingress controller to Service

An Ingress backend names a Service port, not a Deployment. The Service selector then identifies matching pods. A selector such as app: checkout includes only pods carrying that exact label. Confirm the Ingress backend port, the Service port and targetPort, and the port where the process listens. A typo in any one of those fields can produce the same 502 as a dead application.

Service to EndpointSlice to pod

The EndpointSlice controller maintains the eligible backend addresses for a Service, and the Ingress controller uses that data to build its upstream set. EndpointSlices make the selector and readiness result concrete: the expected pod IP and port are either present or they are not. The workflow in this guide to debugging Service endpoints is a useful companion when the selected addresses are unexpected. During termination, an address can remain briefly while the application has already begun draining. The Kubernetes pod lifecycle documentation covers the SIGTERM, grace-period, and SIGKILL sequence that matters here.

Network policy and mesh reachability

Correct labels and current endpoints do not guarantee a usable connection. A NetworkPolicy can allow client traffic but block traffic from the Ingress controller namespace to the backend port. With a mesh, test the proxy path as well as the application path: sidecar health, upstream clusters, connection failures, retry behavior, control-plane connectivity, and mTLS policy compatibility all belong in the same trace. If direct pod access and Service access differ, review the Service forwarding path with this explanation of kube-proxy networking fundamentals. The important question is not “is the pod healthy?” but “which hop first fails to establish or complete the upstream exchange?”

Command-driven diagnosis of a 502

Work from the edge toward the workload. Do not modify several resources at once; that turns a bounded investigation into a new incident.

1. Reproduce the failing request and capture its timestamp

Send the same method, host, and path that users are sending. Record the response code, response headers, request ID if one exists, client location, and UTC timestamp. For an intermittent failure, collect several requests around the incident window and compare them to pod restarts and Deployment events. Testing only from inside the backend pod is not enough—it bypasses the Ingress and network path that may be returning the 502.

2. Read the controller logs for that request

For an ingress-nginx controller, I start with the controller handling the route:

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=10m

Filter by host, path, request ID, or timestamp. Messages such as connection refused, upstream timed out, no live upstreams, and invalid response are more actionable than the client-facing 502. If the controller says it has no live upstreams, move immediately to endpoints. If it says the upstream timed out, compare latency and timeout ownership before increasing a limit.

3. Verify the Service endpoints

Check whether the Service has ready backend addresses and whether its selector and ports describe the workload you expect:

kubectl get endpoints <service-name> -n <namespace>
kubectl describe service <service-name> -n <namespace>

An empty endpoint set usually means the selector matches no pods or readiness has removed every pod from traffic. If endpoints exist but their addresses are terminating or belong to an unexpected workload, correlate that output with the rollout timeline. This is also the answer to the common “does a 502 mean the pod is down?” question: no—an endpoint can be missing, stale, unready, or unreachable while the pod itself still says Running.

4. Check readiness, events, and application logs

Inspect status, node placement, readiness conditions, restarts, probe failures, and scheduling warnings:

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>

Read the Events section, then inspect the application container logs at the same timestamp. A pod can be Running while failing readiness, so Running is not a routing signal. Confirm the probe path, port, startup timing, and response semantics. A readiness probe should test whether the application can serve the request class you are about to send it, not merely whether the process exists.

5. Validate every port and listening address in the route

Trace the port from Ingress backend to Service port, from Service targetPort to the container, and finally to the process. The Ingress references the Service port; the Service target port must resolve to the port exposed by the pod. A process bound only to 127.0.0.1 can make every numeric value look right while remaining inaccessible through the pod network. This check is especially valuable when a Service defines multiple named or numeric target ports.

6. Check NetworkPolicies and mesh boundaries

Review policies in both the Ingress and application namespaces. Confirm that controller pods can reach the backend labels and destination port, and that an egress rule does not block the return path. If sidecars are injected, inspect the sidecar logs and proxy clusters as well. Persistent mesh 502s require a direct check of mTLS compatibility, sidecar health, upstream configuration, connection-pool limits, and control-plane connectivity. Fix the first failing hop; adding retries before that point can amplify the load on a degraded backend.

Fix the first broken boundary

The correct fix is specific to the first failed hop. Once the route is traced, avoid compensating for a selector error with retries or for a slow database query with a larger Ingress timeout.

Correct the Service selector and target port

A Service routes only to pods whose labels match its selector. Confirm that the Ingress names the Service port and that targetPort matches the port where the application listens.

apiVersion: v1
kind: Service
metadata:
  name: registry-api
spec:
  selector:
    app: registry-api
  ports:
    - port: 80
      targetPort: 8080

Compare that selector with kubectl get pods --show-labels, then inspect kubectl get endpoints registry-api. An empty or unexpected endpoint set means the proxy has no valid upstream. Consult the Kubernetes Services and Ingress documentation when reconciling the Service and Ingress fields; this guide to Kubernetes Services configuration is a practical field-level reference.

Gate traffic on readiness, not process startup

A running container is not necessarily ready to serve HTTP. Add a readiness probe for an application health endpoint, and use podReadinessGates when another controller must report a condition before traffic is safe. That is appropriate for workloads that depend on a sidecar, a data load, or a registration event.

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

Probe values should come from observed startup and recovery behavior. A probe that is too shallow sends traffic to a partially initialized process; one that is too aggressive removes healthy-but-busy pods. The pod lifecycle documentation explains the readiness model Kubernetes uses to determine whether a pod receives traffic.

Tune timeouts and retry only safe requests

A long request can exceed the Ingress read timeout even when the backend is healthy. For ingress-nginx, set proxy-read-timeout only after you have compared the Ingress value with the application, service-mesh, and load-balancer limits. Retries can hide transient upstream faults, but I restrict them to idempotent operations such as GET. Retrying a POST without an idempotency strategy can duplicate work.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout http_502"

The ingress-nginx annotation reference documents the timeout behavior. Measure the request’s normal and tail latency before choosing 120; it is an example, not a universal value.

Drain connections during rolling updates

Pods can remain in the routing view briefly while terminating. The application needs time to accept SIGTERM, stop taking new work, and finish active requests before Kubernetes sends SIGKILL. A preStop hook and a sufficient grace period reduce resets during redeployments.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

Do not copy those values without measuring the application’s drain time. The sleep only creates a window; the application still needs real shutdown handling. Kubernetes documents the relevant behavior in its container lifecycle hooks and pod lifecycle guides.

Remove capacity and service-mesh bottlenecks

If the route, endpoints, probes, and shutdown behavior are correct, inspect saturation. An Ingress controller connection pool can exhaust under high concurrency; add replicas behind a load balancer and size CPU and memory from observed load. In a mesh, check mTLS policy compatibility, sidecar health, upstream clusters, connection-pool limits, and control-plane connectivity. A sidecar that cannot establish mTLS can return a 502 before the application sees the request. Istio’s traffic-management documentation is the right reference for policy and routing behavior.

Make 502 prevention part of the platform contract

Prevention is not an annotation you apply after an incident. It is a consistent contract between workloads, Services, proxies, and deployments. Standardize readiness and liveness conventions in shared manifests or policy, give applications enough time to initialize, and use a startup probe when initialization is materially longer than ordinary health checks.

Resource policy belongs in the same baseline. Set requests and limits from observed peak usage, leave headroom for normal concurrency, and monitor restart reasons. An undersized memory limit can OOMKill a backend and leave the Ingress with no usable upstream. At the edge, track controller CPU, memory, active connections, and connection-pool saturation; a fleet with growing traffic may need more Ingress replicas as well as more application capacity.

For every workload, document which layer owns request timeouts and retry policy. Alert on retries instead of treating them as invisible recovery, and test rolling updates against live-like traffic. A 502 SLO that can be correlated with endpoint changes, readiness transitions, restarts, OOMKills, and mesh response details gives operators a shorter path from symptom to cause across clusters.

Technical takeaway: trace a 502 from the proxy log to the exact EndpointSlice address, then fix the first hop that cannot complete the upstream request—do not mask a broken route with restarts, longer timeouts, or retries.