Kubernetes Security Tools: How I'd Build the Stack Layer by Layer
Kubernetes security programs rarely fail because someone picked the wrong scanner. In my experience running platform teams across a lot of clusters, they fail because the controls are scattered. One team runs a scanner in CI, another has half a Gatekeeper install, and nobody can say which layer would have stopped a privileged Pod from reaching production.
So I don't start with a vendor list when I pick kubernetes security tools. I start with the control points the cluster already has, meaning build, admission, runtime, network, identity and audit. Then I give each one a tool, an owner and a clear enforcement mode. This post walks through that stack the way I'd roll it out, with the commands and manifests I'd actually use.
See how Plural helps platform teams secure Kubernetes fleets.
The problem: tools mapped to categories, not control points
Most "security stack" diagrams are vendor categories arranged in boxes. The cluster doesn't work that way. The API server is the front end for the control plane, and every request passes through authentication, authorization and admission before anything is persisted. After that, workloads run on nodes, talk over the network and leave a trail in the audit log. The Kubernetes security documentation covers those native control points: API access control, Pod Security Standards, NetworkPolicies, admission controllers and audit logging.
A tool that scans images but can't tell you who is allowed to create privileged Pods, or what traffic those Pods can reach, covers one slice of the problem. The OWASP Kubernetes Top 10 is a good sanity check here. It lists insecure workload configuration, overly permissive authorization, secrets management failures, missing policy enforcement, weak network segmentation, exposed components, vulnerable components, broken authentication, lateral movement and inadequate logging and monitoring. No single scanner covers that list.
Diagnosis: three questions for every tool
For every tool I evaluate, I ask three things:
- Where does it observe? Source, image, API request, running process or network flow.
- Where can it enforce? Some tools only report. Others block an operation, mutate a resource or feed remediation.
- What evidence does it produce, and who acts on it? A finding with no owner is noise.
Asking these keeps you from treating a manifest linter as runtime detection, or an audit dashboard as a preventive control. It also shows you where the gaps are before you buy something that overlaps with what you already run.
Layer 1: posture and configuration before deploy
This is the cheapest place to catch mistakes, so it comes first. I use two tools here.
KubeLinter scans manifests and Helm charts for common misconfigurations like missing resource limits, containers running as root, privileged containers and writable root filesystems:
kube-linter lint ./deploy/
kube-bench checks a running cluster against the CIS Kubernetes Benchmark. The quickest way to run it is as a Job:
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench
Keep in mind that neither tool changes anything on its own. The findings have to go to someone who will triage them, fix them and track exceptions. On managed control planes, a chunk of the control-plane checks won't apply to you at all, so read the output with that in mind. The Kubernetes CIS Benchmark guide goes deeper on reading those results, and the security posture guide covers turning them into an ongoing posture view.
Layer 2: supply chain in the build pipeline
Next, you need to know what is entering the cluster. Trivy is my default because it covers a lot in one binary: container images, repositories, infrastructure as code and live clusters, checking for vulnerabilities, misconfigurations, secrets and license issues.
In CI I fail the build on fixable high and critical vulnerabilities, and nothing else:
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 registry.example.com/payments-api:1.4.2
trivy config ./deploy/
Why not block on everything? Because a vulnerability result isn't automatically a deployment decision. If every medium CVE with no upstream fix blocks a release, teams start collecting permanent exceptions. Set your thresholds based on severity, exploitability, exposure and whether a fix exists.
Record the image digest and the scan result with the build artifact, and sign the image so you can check where it came from later. With Cosign, verification looks like this:
cosign verify --key cosign.pub registry.example.com/payments-api@sha256:<digest>
Scanning and signing tell you about the artifact. They can't tell you how the workload will behave once it's running. Images also get worse over time as new CVEs land against packages you shipped months ago, so a one-time build gate isn't enough. Rescan what's already running, which is what automated Kubernetes CVE checks are for. There's more on building this into a program in the Kubernetes vulnerability scanning guide.
See how Plural helps platform teams govern secure Kubernetes fleet operations.
See how Plural helps platform teams secure identity and policy across fleets.
Explore how Plural helps platform teams orchestrate secure Kubernetes fleet operations.
Layer 3: admission, where standards become decisions
Admission is where "should" becomes "must." Admission controllers intercept requests to the Kubernetes API and can validate or mutate objects before they're persisted. That makes admission the one place where a policy holds no matter which pipeline or engineer sent the request.
Start with Pod Security Admission
Before installing a policy engine, use what's built in. Pod Security Admission enforces the Pod Security Standards per namespace with labels. To see what would break before you enforce anything, run a server-side dry run:
kubectl label --dry-run=server --overwrite ns --all \
pod-security.kubernetes.io/enforce=restricted
The API server returns warnings for every existing Pod that would violate the profile. Once a namespace is clean, enforce it:
kubectl label --overwrite ns payments \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/warn=restricted
That covers privileged containers, host namespaces, hostPath mounts, extra capabilities and running as root.
Add a policy engine for everything else
For anything the Pod Security Standards don't cover, such as approved registries, required ownership labels or signature checks, you need a policy engine. The two I see most often are Kyverno and OPA with Gatekeeper.
- Kyverno policies are Kubernetes YAML, so most platform teams can read and review them without learning a new language.
- OPA Gatekeeper uses Rego. It has a steeper learning curve, but it's a general-purpose policy engine, and that pays off if you already use OPA outside Kubernetes.
Here's a registry restriction in Kyverno. It's deliberately set to Audit:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-image-registries
spec:
validationFailureAction: Audit
background: true
rules:
- name: validate-registries
match:
any:
- resources:
kinds:
- Pod
validate:
message: "Images must come from registry.example.com."
pattern:
spec:
containers:
- image: "registry.example.com/*"
Gatekeeper does the same thing with enforcementAction: dryrun on a constraint. Either way, run in audit mode first, read the violations, then flip to enforce. Newer Kyverno releases also let you set the failure action per rule, so check which fields your version supports.
The hard part isn't writing policy. It's deciding how much to write. An overly aggressive blocking set breaks legitimate releases and pushes people into permanent exceptions. Start with a handful of high-impact rules, test them against representative workloads, and define the exception path before you need it. The Kubernetes admission controller guide and the Kubernetes security policy tools overview cover the design side in more detail.
Layer 4: identity, RBAC and secrets
Permissive RBAC is the most common finding I see, and most tools don't show it well. kubectl auth can-i is the fastest way to find out what an identity can actually do:
kubectl auth can-i --list \
--as=system:serviceaccount:payments:default -n payments
kubectl auth can-i create pods --all-namespaces \
--as=system:serviceaccount:ci:deployer
If the default service account in an application namespace can read secrets or create Pods, that's your lateral movement path. It won't show up in any image scan.
For secrets, move the source of truth into a dedicated secrets manager so you limit how exposed they are. Also keep the control-plane basics in place: TLS on control-plane traffic and between the control plane and its clients, encryption of control-plane data at rest, and a small, tightly managed set of admin identities. These controls only work if the identity design is sound, keys get rotated and someone reviews access.
Layer 5: network segmentation
A NetworkPolicy only does anything if your CNI enforces it. Cilium and Calico both do. A default-deny policy per namespace is the baseline I start from:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Apply that alone and every Pod in the namespace loses DNS, so you'll immediately need an explicit egress allow to kube-dns:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: payments
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
The tradeoff is that a policy can be technically valid and still break a real dependency. So get a baseline of the traffic you can observe first, then tighten ingress and egress around known service relationships. On Cilium, Hubble makes this much easier:
hubble observe --namespace payments --verdict DROPPED
Review policy changes alongside application changes. Network controls cut down the paths an attacker can reach, but they don't replace identity checks, image validation or application-level authorization.
Layer 6: runtime detection
Static checks can't predict everything. A compromised dependency spawning a shell, or a workload suddenly connecting to a sensitive service, only shows up at runtime. Falco watches system calls and flags that kind of behavior:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco --namespace falco --create-namespace
The default ruleset catches things like a terminal shell spawned inside a container. But is it worth running if nobody's watching it? Not really. Runtime detection only pays off when the rules are tuned against normal deployment behavior, alerts go to an owner with a defined severity model, and there's a documented response for suspicious process execution or unexpected egress. If your pilot produces more untriaged alerts than the team can review, stop and tune before you expand. The Falco runtime security guide goes further into rule tuning.
Layer 7: audit logs as evidence
Kubernetes audit logs record, in order, the actions taken by users, API clients and the control plane. They're how you prove a policy decision happened, trace a change and document remediation. On self-managed control planes you pass the API server an audit policy with --audit-policy-file and a destination with --audit-log-path. A reasonable starting policy logs RBAC changes in full and only records metadata for secrets, so secret values never end up in your logs:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
resources:
- group: "rbac.authorization.k8s.io"
resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
- level: Metadata
Managed providers expose audit logs through their own logging services. Either way, send the logs to your SIEM next to your scanner and runtime findings, so platform and security teams can investigate from the same data.
How I evaluate a tool before it goes in the stack
Once the layers are clear, evaluating a tool is mostly about proving it works in your environment:
- Trace a real finding end to end. Take a representative finding and follow it to an owner, a ticket, a policy exception and a fix. Check how the tool removes duplicates, rates severity, identifies the affected workload and separates exploitable issues from theoretical ones.
- Check it against independent baselines. Compare the vendor's claims with kube-bench output and your own audit logs. For container-specific risk modeling, NIST SP 800-190, the Application Container Security Guide, is useful context even if you aren't bound by it.
- Test portability. Run the same policy on every distribution and cloud you support, plus any disconnected environments. Policies that depend on provider-specific metadata or an always-on SaaS endpoint will fail in private, hybrid or air-gapped clusters.
- Measure fleet-scale cost. Look at agent resource usage, scan duration, API server load, upgrade behavior and how much work it takes to onboard the next cluster.
Also write down what the tool doesn't cover. That list is how you find the next gap. For the compliance side, the compliance as code tools roundup is a useful comparison point.
What changes at fleet scale
With one cluster, everything above is a weekend of Helm installs. With fifty clusters across regions, clouds and a couple of air-gapped sites, the hard part becomes distribution. How do you get the same Kyverno policies, NetworkPolicies, Falco rules and exception records onto every cluster, and prove they're still there next month?
I'd centralize policy intent and evidence, but keep execution close to the workloads. A management plane that stores admin credentials for every cluster and pushes changes over inbound connections becomes the most valuable target in your whole estate.
This is the problem Plural is built for, so it's worth being specific about what it does and doesn't do. Plural runs thin deployment agents on each managed cluster. The agents open egress-only connections to a self-hosted control plane and apply changes with local cluster credentials, so those credentials never have to be stored centrally. You keep policies, scanner config and alert routing in Git, and GitOps-based continuous deployment promotes reviewed changes across environments with drift detection. Infrastructure-as-code management coordinates the Terraform, Pulumi or Ansible changes that go with them. The same model works for cloud, on-prem, edge and air-gapped installs. In an air-gapped site, though, you still have to design image provenance, software transfer, patch windows and local monitoring yourself.
What it doesn't do is replace the security tools. Trivy still scans, Kyverno or Gatekeeper still enforces at admission, and Falco and your CNI still produce runtime and network signals. The fleet layer distributes approved configuration and records how it reached each cluster.
A rollout order that holds up
- Inventory clusters, namespaces, registries and owners. If a production workload has no owner, stop. Its alerts and exceptions will have nobody to answer them.
- Run a baseline. Run KubeLinter, Trivy and kube-bench against representative repos, charts, images and clusters before you change any enforcement. Note what keeps coming up: privileged containers, missing limits, exposed services, embedded secrets.
- Enforce a small policy set in audit mode first. Pick restricted Pod Security, approved registries, ownership labels and namespace default-deny. Move them from audit to enforce in a non-production cluster before production. Every exception record needs an owner, a reason, a compensating control and an expiry date, because an exception with no expiry is just a permanent bypass.
- Add runtime and network detection where someone will act on it. Tune rules against known behavior and decide which events page on-call.
- Distribute everything through GitOps. That means policies, scanner config, exceptions and alert routing, all promoted with pull requests. The compliance automation guide shows how to turn this into repeatable evidence.
- Review monthly, then expand. Look at policy coverage, critical findings by age, exception counts and overdue exceptions, image remediation time, runtime alert precision and the share of clusters reporting evidence. Only move to the next wave when the current stage hits its target.
Conclusion
The best kubernetes security tools won't help you if they don't line up with the cluster's actual control points. Linters and scanners catch cheap mistakes early. Admission enforces the few things that must never ship. RBAC, NetworkPolicies and Falco limit what an attacker can reach and flag what they do, and audit logs prove all of it happened.