Infrastructure as Code with Terraform: How to Run It Safely at Enterprise Scale

Getting started with infrastructure as code in Terraform is easy. You write some HCL, run terraform apply, and a VPC shows up. The hard part comes later, once dozens of engineers, several cloud accounts, and a fleet of Kubernetes clusters all depend on the same state. Then you're dealing with a plan nobody reviewed, a state file two pipelines tried to write at once, or a console hotfix that quietly gets reverted by the next apply.

In my experience running Terraform behind Kubernetes fleets, nearly all of the pain comes from a few decisions teams make early and rarely revisit: what Terraform should own, how state is split and protected, how plans get reviewed, and how drift gets caught. This post goes through each one as a problem, a diagnosis, and a fix.

The Model: Declarative Config, Tracked State, Inspectable Change

The basic idea is simple. You describe the resources you want in declarative configuration, and Terraform works out the changes needed to move real infrastructure toward that description. You're declaring what should exist, not scripting a sequence of operator actions. If you want the formal version, HashiCorp's Terraform introduction covers it.

Providers connect that configuration to the outside world. A provider turns Terraform resources into API calls against AWS, GCP, Azure, Kubernetes, Helm, or whatever else you're targeting. Because providers expose the relationships between resources, Terraform can build a dependency graph and figure out the right order to create and destroy things.

The core loop is short, and every step can be inspected:

terraform init            # install providers, configure the backend
terraform plan -out=tfplan  # diff config + state against live infrastructure
terraform apply tfplan      # execute exactly the plan you reviewed

The piece people underestimate is state. It isn't a cache. It's how Terraform maps resource addresses in your config to real objects in your cloud account. If you lose it, Terraform no longer knows what it manages. If you corrupt it, the next plan can be badly wrong. If you leak it, you may have leaked secrets. Putting your config in Git doesn't solve any of that. You need a deliberate model for state and for who can access it.

Problem 1: Terraform Is Managing Things It Shouldn't

Diagnosis

The most common failure I see is a team that picks up the Kubernetes and Helm providers and starts managing everything through Terraform, down to individual Deployments. It works for a while. Then every application release needs a Terraform plan, one bad workload change blocks a network change in the same root module, and the blast radius of a routine deploy now reaches your cloud infrastructure.

People frame this as Terraform versus Kubernetes, but the actual boundary is infrastructure lifecycle versus workload lifecycle.

Fix: draw an explicit ownership line

Here's the split I recommend:

Terraform owns cloud accounts, networking, identity and IAM, managed data services, cluster control planes, node groups, and other foundational dependencies. These change slowly and deserve a plan/approve/apply cycle.
Kubernetes delivery owns namespaces, workload manifests, Services, Ingress, policies, and Helm releases. These change often and belong in a GitOps-based continuous deployment loop that reconciles them inside the cluster.
The platform team owns the interface between the two. That means deciding which Terraform outputs get exposed to workloads and which changes need coordinated review.

Take a setup with production clusters in two regions and a separate dev cluster. Terraform creates the network boundaries, private endpoints, identity bindings, cluster versions, and node pools for each environment, and its plan will show you that a network change has to land before a cluster that depends on it. After that, your delivery system installs platform add-ons and ships applications without re-planning cloud infrastructure on every release.

The handoff is where the real work is. A Terraform output might be a database endpoint, a bucket name, or a workload identity reference that some service consumes:

output "orders_db_endpoint" {
  value       = aws_db_instance.orders.address
  description = "Consumed by the orders service in the prod clusters"
}

Document and validate that contract. It shouldn't become an excuse for either system to take over the other's domain. For the wider operating context, see this guide for Kubernetes platform engineering teams.

Problem 2: One Giant State File (or a Pile of Unversioned Modules)

Diagnosis

Terraform structure is an ownership problem as much as a code organization problem. You can spot the symptoms quickly:

A single root module whose plan touches dev, staging, and prod together
terraform plan runs that take minutes because they refresh hundreds of unrelated resources
Modules pulled from main with no pinned version, so a refactor in one repo quietly changes plans everywhere
Engineers who have prod state access only because dev and prod share a backend

Each of these makes your blast radius bigger. When a plan fails or a state entry needs manual repair, you want the damage limited to one environment and one team.

Fix 1: keep modules focused and versioned

A reusable module should wrap one coherent capability, like a network, a managed database, or a cluster foundation. It should expose deliberate inputs and outputs instead of passing through every provider argument. The calling configuration supplies environment-specific values such as names, regions, and sizes. That's how you use one template to build consistent dev, test, and prod environments while still treating their risk and access requirements differently.

Pin module versions and treat upgrades as reviewed changes:

module "cluster" {
  source = "git::https://github.com/acme/terraform-modules.git//eks-cluster?ref=v1.4.0"

  name        = "prod-us-east-1"
  node_groups = var.node_groups
}

If you publish to a registry, use a version constraint (version = "~> 1.4"). Commit .terraform.lock.hcl too, so provider versions don't drift between laptops and CI.

Repository boundaries should follow ownership and lifecycle. The platform team might own the cluster and network modules, while app teams consume versioned interfaces from separate environment repos. Keep each root module small enough that its plan describes one change a reviewer can understand. I'd also avoid clever for_each constructions that create or destroy unrelated resources in a single pass. A clear one-to-one mapping between a resource and its Terraform address is much easier to review, and much easier to recover with terraform state mv or a moved block when you refactor.

Fix 2: give every environment its own remote, locked state

Shared production workflows need remote, access-controlled state. HashiCorp's Terraform state documentation explains the model. I won't tell you which backend to use, because that depends on your cloud and your regulatory requirements. Whatever you choose needs three properties: shared access for authorized automation, locking so two writers can't collide, and an audit trail.

On AWS, a per-environment S3 backend looks like this:

terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "network/us-east-1/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # S3-native locking on recent Terraform versions
  }
}

(Older setups use a dynamodb_table for locking instead. Either way, confirm locking is actually on.) When a crashed CI job leaves a lock behind, terraform force-unlock <LOCK_ID> exists, but treat it like a production change: first confirm nothing else is running.

Doesn't terraform workspace handle environment separation? Only partly. CLI workspaces give you separate state files under the same backend configuration, which is fine for short-lived feature environments. They don't give you separate credentials or separate access control, so I don't use them as the wall between dev and prod. Separate backends, or at least separate buckets and keys with separate IAM, do that job.

Also keep in mind that values marked sensitive are still stored in state. Protect state access as carefully as the credentials it might contain. For more on credential handling and exposure risks, see these IaC security best practices.

Problem 3: Plans Get Applied That Nobody Actually Reviewed

Diagnosis

A reviewer who reads the HCL diff hasn't reviewed the change. The change is the plan. A two-line diff to a variable can force a database replacement. If your workflow is "approve the PR, then someone runs terraform apply from their laptop," the plan that actually executes was never seen by anyone else, and it was generated against whatever credentials and Terraform version that engineer had installed.

Fix: PR-based plan review with a single apply

This is the production workflow I'd set up. The broader operating model is covered in this guide to GitOps for infrastructure as code.

Open a focused pull request. Keep it narrow. Name the environment, the affected resources, expected dependency changes, the owner, and rollback considerations. A clean local run doesn't count as approval to change shared infrastructure.
Run static checks in CI using the repo's pinned Terraform version and lock file:
```
terraform fmt -check -recursive
terraform init -input=false
terraform validate
```
These catch malformed config and provider problems before a human spends time reviewing something that can't work.
Generate the plan against the real backend and post a readable summary on the PR:
```
terraform plan -input=false -out=tfplan
terraform show -no-color tfplan > plan.txt
```
Read the additions, updates, replacements, and destroys, not just the Plan: X to add, Y to change, Z to destroy line. An unexpected -/+ replacement should stop the pipeline until someone understands the cause.
Evaluate policy against the plan, not the source. Convert the plan to JSON so OPA or another policy engine gets structured input:
```
terraform show -json tfplan > plan.json
```
Write policies for approved regions, network exposure, encryption, IAM scope, tagging, and destructive actions. Remember that OPA only sees what's in the plan. It can't see unmanaged resources or changes made outside Terraform, so you still need separate controls for out-of-band changes.
Require approval from the right people. The resource owner signs off on the plan, plus security or platform reviewers where it makes sense. Require explicit confirmation for production, destructive actions, privilege changes, and state migrations. Record the approvers, plan artifact, commit, Terraform version, and policy result.
Apply the reviewed plan exactly once, from CI, using the saved artifact:
```
terraform apply -input=false tfplan
```
Don't regenerate a fresh plan at apply time. That new plan is one nobody reviewed. Scope the runner's credentials to what this root module needs. In regulated or air-gapped environments, keep credentials inside the execution boundary rather than in a central store.
Verify and assign ownership. Confirm the apply finished, outputs have the expected values, dependent Kubernetes services are healthy, and monitoring shows no regression. Link the state and run logs to the change record. The team that owns the resource also owns drift follow-up.

Wiring all of this up for one repo is manageable. Wiring it up consistently for every stack across a fleet of clusters is where most teams end up with a patchwork of CI templates. This is where Plural's IaC management helps: you register existing Terraform repos as stacks through the API without migrating them, commits trigger runs, and pull requests get a plan with automated comments for reviewers. It runs through an agent-based pull model that only needs egress networking and executes with local credentials, including in air-gapped deployments, so the "keep credentials in the execution boundary" rule holds without extra effort.

Problem 4: Drift You Don't Find Until the Next Apply

Diagnosis

Drift is a day-2 problem. It's the gap between the infrastructure that actually exists and the configuration and ownership model you approved. It usually starts with a console change during an incident, an emergency IAM tweak, or someone "just fixing" a security group. If you can't see the gap, the next routine plan can trigger a surprise replacement or undo a deliberate operational fix.

Terraform does detect drift, but only when terraform plan runs against live infrastructure. Nothing is watching continuously. If nobody plans a stack for three months, its drift has gone unnoticed for three months.

Fix: scheduled plans, classification, and clear reconciliation owners

Plan on a schedule, not only on PRs. For important environments, run a periodic plan with a clear exit signal:
```
terraform plan -detailed-exitcode -input=false
# exit 0 = no changes, 1 = error, 2 = changes present (drift or unapplied config)
```
If you want to see only what changed outside Terraform, without proposing config changes, use terraform plan -refresh-only.
Treat drift output as a signal, not an auto-apply trigger. A changed security group, cluster setting, or network dependency needs an owner who decides which of three things it is: unauthorized drift to revert, an approved exception to record, or a real change that belongs in Git. This deeper look at Terraform drift detection tools covers the detection layer and its tradeoffs.
Don't let two controllers own the same field. Terraform reconciles the infrastructure underneath a cluster. Kubernetes controllers reconcile the objects inside it. If Terraform sets a Deployment's replica count while an HPA also manages it, you've built a fight between them. Fix drift by restoring the declaration of whichever system owns that field, not by applying whatever the tool that noticed first suggests.
Make exceptions expire. Record the reason, duration, and owner. Otherwise a temporary manual change becomes permanent and undocumented.

Fleet visibility is the real day-2 problem

With one stack, a green plan tells you something. With hundreds of stacks across a fleet, it tells you very little. You need to know which stacks were planned recently, which are showing differences, and which are waiting on an owner. Plural's Terraform state diagrams and real-time state indexing help here: you can search and trace relationships across infrastructure and Kubernetes services, and sensitive-data pruning keeps raw secrets out of the index. Plural can also export Terraform outputs directly to Kubernetes services, so the handoff from Problem 1 becomes a tracked link and not a value someone copied into a Helm values file. It all sits next to your existing Terraform, Helm, and Kustomize workflows through a unified control plane that you self-host on a management cluster, kept separate from your workload clusters.

For enterprise Kubernetes platform engineering teams, the loop to aim for is detect, classify, assign, remediate, and verify. The same loop handles routine upgrades and incident follow-up, and it leaves the audit trail regulated environments require.

What About Pulumi, Ansible, or Cloud-Native IaC?

Is Terraform the right choice at all? It depends on your cloud footprint and your team. Terraform's model of declarative HCL, a state file, and a plan/apply workflow is a good fit for multi-cloud infrastructure lifecycles. Pulumi keeps a similar desired-state model but uses general-purpose languages. Ansible is mostly procedural configuration and orchestration. Cloud-native tools usually target a single provider. Many enterprises run more than one of these, and that's fine as long as each has a clear ownership boundary and goes through the same review discipline.

Conclusion

Terraform's syntax isn't where things get hard at scale. The difficulty comes from blast radius, state protection, plan review, and drift, and each of those depends on an ownership decision you either make on purpose or end up with by accident. Draw the infrastructure/workload line clearly, split state per environment and owner, pin your modules, apply only the plans people reviewed, and schedule plans so drift can't pile up unnoticed.

The takeaway: in infrastructure as code with Terraform, the plan is the change. Generate it in CI against locked remote state, review it on the pull request, and apply that exact artifact once. Every other practice in this post is there to make that one rule hold across your whole fleet.