Kubernetes DNS resolution failed: a command-driven debug path

When a workload can't resolve a Service, the error in the app is almost never the useful part of the incident. It might be a typo, or the wrong namespace, or CoreDNS actually being down, or a default-deny NetworkPolicy that dropped UDP/53 — and from the client's side those look identical. Start by identifying the failure boundary before you change anything cluster-wide.

You're probably wondering why this needs a tree at all. Seems like overkill for nslookup, right? The problem is that cluster DNS sits on top of Services, kube-proxy, the CNI, and whatever upstream resolver the node was given, and a successful A record does not mean the app can connect. The Kubernetes DNS debugging guide is the right reference; this is the order I actually run the commands in.

There are four checks, ranked from cheapest / lowest blast-radius to "now you're on the node and the upstream":

• Can a disposable Pod in the affected namespace resolve a known Service?
• Are the CoreDNS Pods running, and does this Pod's /etc/resolv.conf even point at them?
• Does the Service exist, and does it have endpoints? That's a separate question from DNS.
• Does only external DNS fail, or does the same lookup fail from the worker node?

If you skip ahead to rewriting the Corefile, you'll "fix" an upstream outage by breaking in-cluster search domains. Don't.

Start with a disposable Pod

Run this in the affected namespace, swapping app-namespace and api for real values. The namespace matters: the search list kubelet writes into the Pod is namespace-specific, so a test Pod in default will not reproduce a failure in payments.

kubectl run dns-check -n app-namespace --rm -it --restart=Never --image=busybox:1.36 -- nslookup api

Kubernetes' own guide uses a dnsutils Pod with agnhost; busybox:1.36 is the same check with a smaller image.

If that returns an address, cluster DNS is answering. The app may have its own resolver, a hard-coded hostname, or it's looking in the wrong namespace — a short name is namespace-relative, so a Pod in payments asking for api is asking for api in payments, not api in platform. Use api.platform or api.platform.svc.cluster.local for the cross-namespace case. Kubernetes documents this in DNS for Services and Pods; there's a longer writeup of the discovery model if you want the Service-side of it.

If you get server can't find, a timeout, or it can't reach the nameserver, stay on the CoreDNS branch.

Are the CoreDNS Pods actually up?

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

The label is k8s-app=kube-dns even when CoreDNS is the implementation — that's not a typo, and the Service is usually named kube-dns too. CoreDNS is the cluster DNS on current clusters; older ones may still be running kube-dns. If Pods are Pending, CrashLoopBackOff, or not Ready, inspect scheduling and config before you touch the application again. Running is not the same as answering, so read the logs for processing errors and upstream timeouts.

Then look at what the test Pod is actually using:

kubectl run dns-shell -n app-namespace --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf

You want a nameserver pointing at the cluster DNS Service, search domains for the Pod's namespace plus the cluster suffixes, and options ndots:5. A missing or unexpected nameserver is a Pod-level dnsPolicy / dnsConfig issue, or the node's kubelet --resolv-conf, not a broken Service object.

The ndots:5 default is a forcing function for the weird lookups you'll see in packet captures. A name with fewer than five dots is not treated as fully qualified, so api.platform still gets the search list appended (api.platform.svc.cluster.local, then api.platform.cluster.local, and so on) before the stub gives up. If you actually mean the FQDN, use the trailing dot. I mention this because a lot of "DNS is slow" tickets are the search list, not CoreDNS being overloaded.

Does the Service exist, and does it have endpoints?

kubectl get svc api -n app-namespace
kubectl get endpoints api -n app-namespace
kubectl get endpointslices -n app-namespace -l kubernetes.io/service-name=api

If the Service is missing, the name is wrong or it's in another namespace. If it exists with no endpoints, that's a selector or readiness problem — DNS can still return the Service's cluster IP. A resolved name is not a reachable app. The endpoints walkthrough covers the next checks; don't "fix DNS" because get endpoints was empty.

Does only external DNS fail?

kubectl run dns-external -n app-namespace --rm -it --restart=Never --image=busybox:1.36 -- nslookup registry.corp.local

If Service names resolve but registry.corp.local doesn't, the cluster DNS path is working. Look at CoreDNS forwarding, upstream reachability, firewalls, and resolver limits. If both fail, go back to CoreDNS, resolv.conf, and NetworkPolicy. Policies have to permit DNS to the cluster DNS Service on port 53 — UDP, and TCP where the query falls back.

CoreDNS and resolv.conf

When the app itself is the one failing, inspect that Pod's resolver before you edit CoreDNS:

kubectl exec -n <namespace> <pod> -- cat /etc/resolv.conf

Same expected contents as the test Pod: nameserver → cluster DNS, search list for the namespace, ndots:5. The search list is why api resolves inside the Pod's namespace and why a Service in another namespace needs api.platform or the FQDN. Cross-namespace communication is mostly this naming rule plus whatever NetworkPolicy you've added on top.

Then check the DNS workload, the deployment, and the Service together:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get deployment -n kube-system coredns
kubectl get service -n kube-system kube-dns

If the deployment has no ready replicas, inspect events and logs rather than treating every lookup failure as an application problem:

kubectl describe deployment -n kube-system coredns
kubectl logs -n kube-system deployment/coredns

Logs will show processing errors, plugin failures, or forwarding timeouts. If internal Service names fail and CoreDNS logs show no corresponding requests, the query never arrived — that's the path to the DNS Service (NetworkPolicy, CNI, node). If CoreDNS receives the request, resolves internal names, and external names time out, leave the kubernetes plugin alone and look at forward / proxy plus egress.

Read the Corefile:

kubectl get configmap -n kube-system coredns -o yaml

Look at the kubernetes plugin for cluster zones, and forward (or the older proxy) for upstreams. A broken upstream takes out external domains. A malformed cluster-zone config, or CoreDNS losing API access, takes out Service and Pod records. Change configuration only after you know which class of lookup fails. In particular, an upstream outage is not fixed by rewriting Pod resolver settings.

When DNS is working and the app still isn't

A lookup can succeed while the path to the requested Service is incomplete. Check the object, the selector, and the endpoint data before you treat it as CoreDNS.

kubectl get svc -n <namespace> <service> -o wide
kubectl describe svc -n <namespace> <service>
kubectl get endpoints -n <namespace> <service>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>

Compare the selector from describe with the labels on the intended Pods. Then check whether the EndpointSlices contain ready addresses and the port the client actually uses. An empty endpoint set points at workload readiness, selector, or port configuration. It does not, by itself, mean DNS resolution failed.

If the Service has endpoints, the next place traffic dies is node-level forwarding. kube-proxy programs iptables or IPVS to turn a Service VIP into backend Pod IPs. A kube-proxy error, a stale ruleset, or a node-specific problem can make a resolved IP unusable. Compare the same lookup and connection from Pods scheduled on different nodes, and read kube-proxy logs when the symptom follows one worker.

NetworkPolicy is the other common boundary. A default-deny egress policy must allow UDP and, where required, TCP on port 53 to the DNS Pods in kube-system. Check ingress on the DNS deployment too. And test from the affected workload — a successful query from an unrestricted debugging Pod tells you almost nothing about a policy that selects the app.

Finally, look at the CNI when Pod IPs, routes, or cross-node traffic are inconsistent. The CNI is what DNS packets actually travel on. Keep the conclusion precise: DNS proves name resolution. It does not prove endpoint readiness or that the application answered.

Node vs upstream

Run the same lookup from two failure domains: a disposable Pod, and the worker that hosts the affected one.

kubectl run dns-test --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default
kubectl run dns-test --rm -it --restart=Never --image=busybox:1.36 -- nslookup registry.corp.local

If the cluster name fails in one Pod and succeeds in another, it's that Pod's namespace, policy, network identity, or /etc/resolv.conf — not a cluster-wide CoreDNS outage. If every Pod fails, inspect CoreDNS before you go down the stack. A Pod failure paired with a successful lookup from its worker usually means the in-cluster path (CNI, policy, kube-proxy). Microsoft's AKS notes document the same split: failure from the Pod but not the node is typically cluster networking.

What a node test actually proves

Identify the worker hosting the affected Pod, then test from that node:

kubectl get pod <pod> -o wide
nslookup kubernetes.default.svc.cluster.local
nslookup registry.corp.local

A successful node lookup and a failed Pod lookup rules out a simple upstream outage on that node. Focus on the CNI route to the cluster DNS Service, NetworkPolicy on port 53, and node-level forwarding. A node-wide failure — unrelated Pods on the same worker also failing — raises the odds of a node firewall, a cloud security group, a broken route, or node-local DNS config.

When the upstream is actually the problem

If kubernetes.default resolves and registry.corp.local times out, CoreDNS can answer cluster records while forwarding is unhealthy. Check the Corefile and logs for forwarding errors, timeouts, or exhausted resolver capacity. Then check that the node firewall and security group permit DNS egress to the configured upstreams. If external lookups fail from Pods and nodes across the cluster, the upstream (or the path to it) is the incident. If internal names fail too, stay on CoreDNS Service reachability, CNI, and port 53.

Some clusters put a node-local DNS cache in front of CoreDNS. In that design, compare the nameserver in the Pod's resolv.conf with the node-local listener and confirm it forwards to CoreDNS. A mismatch fails selected nodes. A shared upstream outage fails everyone.

What the result actually tells you

The thing this path doesn't do

This whole sequence is cluster-local. That's fine until you have more than one cluster and the same Corefile tweak or NetworkPolicy ships inconsistently. The commands don't change; you just notice you're repeating them, and the useful question is whether the CoreDNS ConfigMap and the policy that landed with it actually match.

If those clusters are already registered with a GitOps controller, you can compare the objects without SSHing around. Plural's agent is pull-mode for this — the management plane isn't punching into the cluster, the agent polls — so you're looking at the same ConfigMap revision the controller thinks it applied. That still doesn't repair DNS, and it doesn't tell you the upstream is healthy. It just keeps you from treating five identical Corefiles as five independent CoreDNS outages.

Record the failing cluster, namespace, query, and timestamp. Promote the verified Corefile and policy through the same delivery path you use for everything else. There's no universal right answer on node-local cache vs central CoreDNS vs more replicas; the forcing function is usually whether the failure is one Pod, one node, or the upstream.

That's it. If you want a single command to come back to:

kubectl run dns-check -n app-namespace --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default

If that works and the app still fails, it isn't cluster DNS. If it doesn't, you know which branch you're on.