Terraform Drift Detection — Finding and Fixing Infrastructure That Changed

Overview and What You Will Learn

Drift happens when an engineer logs into the AWS console at 2am during an incident and manually bumps a security group rule, or widens an RDS instance's storage, just to make the immediate problem go away. The fix works. The incident closes. But now your Terraform configuration still describes the OLD rule, and your Terraform state still believes the OLD rule is what is deployed. Reality moved. Terraform's understanding of reality did not. That gap is drift — and left unmanaged, it quietly grows until nobody trusts terraform plan output anymore because it is always full of unexplained changes nobody asked for.

By the end of this topic you will be able to:

• Explain exactly how and why drift happens, with real incident-response scenarios
• Use terraform plan as your first and fastest drift detection tool
• Understand what -refresh-only does and how it differs from a full apply
• Set up scheduled, automated drift detection in CI
• Use Driftctl as a dedicated, more comprehensive drift detection tool
• Decide, case by case, whether to reconcile drift back to your config or update your config to match reality
• Prevent drift at the source with IAM policies that block console changes on Terraform-managed resources
• Follow a clear incident response procedure the next time drift happens in production

Why This Matters in Production

At Hotstar, during a high-traffic cricket match, an on-call engineer noticed an ALB target group was returning unhealthy targets and manually changed the health check path directly in the AWS console to stop the bleeding — a completely reasonable, correct call under incident pressure. Three weeks later, a routine terraform plan on an unrelated PR showed an unexpected change to that same health check path, with no explanation in the diff. The reviewer almost approved it as probably fine before someone traced it back to the 2am incident fix that nobody had ever told Terraform about. Without that trace, the PR would have silently reverted a deliberate, still-needed fix back to the old broken health check — right before the next big match.

Drift is not just a technical problem. It is a trust problem. When your team stops believing terraform plan output, you have lost the most important safety net in your infrastructure workflow.

Core Principles

The Drift Lifecycle

What Drift Is Not

Drift is not the same as a bug in your Terraform code. Drift means real infrastructure diverged from what Terraform believes is deployed. Your code might be perfectly correct — the problem is that someone changed reality without going through Terraform.

Remember

terraform plan always refreshes its view of real infrastructure before comparing to your config. Every time you run plan, you are also running drift detection — whether you intended to or not.

Detailed Step-by-Step Practical Lab

Step 1 — Detect Drift with terraform plan

terraform plan is your default drift detector. It reads real infrastructure state from the cloud provider API before comparing anything:

The phrase "Terraform detected the following changes made outside of Terraform" is explicit. Terraform is telling you: this diff was not caused by your code — reality moved without you.

Tip

Read every plan output top to bottom before approving. The drift section appears ABOVE the planned changes section. Engineers who skim plans often miss it entirely.

Step 2 — Update State Only, Without Changing Infrastructure

If the manual change was intentional and correct — like the Hotstar health check fix — use -refresh-only to acknowledge it in state without touching real infrastructure:

Security

Running -refresh-only without then updating your .tf file leaves your code out of sync with state. The next engineer who reads the .tf file will see the wrong config. Always follow a -refresh-only apply with a matching .tf update and PR.

Step 3 — Reconcile Drift Back to Your Configuration

If the manual change was wrong, or a temporary emergency fix you do not want to keep, a normal terraform apply reverts reality back to match your code:

Step 4 — Automate Scheduled Drift Detection in CI

Do not wait for an unrelated PR to stumble onto drift. Run a scheduled read-only plan against production every day:

Tip

Create a read-only IAM role specifically for the drift detection job. It only needs terraform plan permissions — not apply. This limits blast radius if the CI credentials are ever compromised.

Step 5 — Use Driftctl for Deeper Detection

terraform plan only detects drift on resources that exist in your .tf files. Driftctl also finds resources that were created in your AWS account and never touched by Terraform at all:

Common Mistake

Assuming terraform plan gives you the complete drift picture. It only compares resources that exist in your state file against reality. An EC2 instance someone clicked into existence and never imported into Terraform is completely invisible to terraform plan — but Driftctl surfaces it.

Step 6 — Prevent Drift with IAM Policies

The strongest long-term fix for drift is making direct console modifications impossible for Terraform-managed resources:

This policy denies modification of any resource tagged ManagedBy: terraform to anyone except the GitHub Actions role. Console engineers, direct CLI access, and even other automation tools are blocked. Only the CI pipeline can change these resources.

Tag your Terraform resources consistently so this policy covers everything:

Step 7 — The Drift Incident Response Procedure

Follow this every time drift is discovered — whether through a scheduled job alert or a confusing PR plan:

Remember

During an active incident, stop the incident first. Preserve Terraform consistency second. Make the manual fix, then follow this procedure once the incident is resolved. Never let Terraform purity get in the way of restoring service.

Production Best Practices and Common Pitfalls

• Never let drift sit for more than one sprint. The longer a manual change goes un-reconciled, the more likely someone will revert it accidentally — or build on top of it without knowing it is drifted.

• Treat -refresh-only as half the job. Updating state without updating the .tf file means the next plan will show the same drift again. Always follow up with a code change.

• Run Driftctl on a schedule. Unmanaged resources accumulate silently over months and years. One-time scans only show today's problem.

• Add lifecycle { ignore_changes = [...] } for auto-managed attributes. Some AWS resources self-modify legitimately — like Auto Scaling groups adjusting desired_capacity or AMIs being rotated by automation. These will appear as drift on every plan. Use ignore_changes to tell Terraform these attributes are intentionally managed outside Terraform.

• Combine prevention with detection. IAM policies prevent most console drift. Scheduled detection catches the rest — including emergency break-glass changes and changes made by other teams' automation.

Quick Reference and Troubleshooting Commands