)
)
With Terraform and OpenTofu, you define your infrastructure declaratively, describing how you want the infrastructure resources configured. Both tools then take the necessary steps to ensure the deployed infrastructure matches the desired state when running commands such as terraform apply and tofu apply .
However, once you have deployed some infrastructure resources, you can alter them through various methods, keeping the infrastructure dynamic and open to changes beyond Terraform’s and OpenTofu’s, which can cause misalignment.
This misalignment is called drift and can lead to security vulnerabilities, cost explosions, degraded pipelines and other risks. Therefore, detecting and managing drift effectively is essential for teams using Terraform and OpenTofu.
In this guide, we will explore why drift happens, the risks caused by drift, and the options available to detect, manage and remediate drift.
Here’s a quick overview of what we will cover:
- State: How Terraform and OpenTofu track resources
- What is Drift in Terraform and OpenTofu
- What are the Most Common Reasons for Drift in Terraform and OpenTofu
- The Impact and Risks of Drift
- Drift in Terraform and OpenTofu: A Practical Example
- How to avoid Drift in the first place
- How to Detect and Manage Drift
- How to Detect, Manage and Remediate Drift in Terraform and OpenTofu with Terramate
State: How Terraform and OpenTofu track resources
Let’s start by explaining how the state in Terraform and OpenTofu works. Both tools store information about the managed infrastructures and configuration in state files, which are essential to both tools and perform the following functions:
- Map resources defined in Terraform and OpenTofu configurations with real-world resources.
- Track metadata about resources such as dependencies and dependency order.
- Cache resource attributes to improve performance when managing infrastructures at a scale.
- Syncing, which enables better collaboration among teams.
- Track resources managed by Terraform and OpenTofu to separate and ignore other resources in the same environment.
The state holds all important details about resources created or managed by Terraform and OpenTofu, including information such as resource type, name, provider, configuration, and present state. Both tools rely on this state file to track infrastructure changes and ensure everything aligns with your desired configurations.
Think about the state in Terraform and OpenTofu as the state of the last deployment ran with terraform apply or tofu apply .
The state is persisted in JSON files that are meant to be touched by Terraform and OpenTofu exclusively. Modifying the state files manually is highly discouraged. Instead, use commands such as terraform show or tofu show to show the current state of your current Terraform or OpenTofu root module. In the following examples, we will focus on OpenTofu, but you can easily interchange all commands with terraform also.
$ tofu show
aws_instance.example:
id = i-022g91123egf35efh9
ami = ami-727be440
availability_zone = us-east-1d
instance_state = running
instance_type. = t1.micro
tags.drift_example = v1
tags.name = ec2-drift
...You can also use tofu state show (or terraform state show ) to inspect a specific resource:
$ tofu state show aws_instance.example
id = i-022g91123egf35efh9
ami = ami-727be440
availability_zone = us-east-1d
instance_state = running
instance_type = t1.micro
tags.drift_example = v1
tags.name = ec2-drift
...What is drift in Terraform and OpenTofu?
Drift is the term for when the real-world state of your infrastructure differs from the state defined in your configuration. It is usually caused by manual changes (e.g., changing infrastructure in a cloud provider's UI), automation or external tools.
It’s essential to understand that Terraform and OpenTofu cannot detect the drift of resources that aren’t managed by Terraform and OpenTofu. For example, if you deploy an EC2 instance with OpenTofu or Terraform and then use Ansible to install a few software packages in the machine's OS, Terraform won’t detect those changes. However, if you log in to the AWS Cloud Console and change the tags associated with the same EC2 instance, OpenTofu and Terraform can detect that change as a drift made outside your IaC configurations.
Terraform maintains a state file to track your managed infrastructure and configuration.
Drift occurs when changes are made outside of your IaC configuration.
What are the most common reasons for drift in Terraform and OpenTofu?
Now that we have learned what drift is, let’s look at some of the most common causes of drift and how those can be prevented.
1. Manual changes outside of Terraform
Manual changes are often the primary cause of infrastructure drift and are almost always caused by developers making changes directly through cloud consoles (the cloud providers’ UIs, such as the AWS Console) or CLI tools instead of going through infrastructure as code. When resources are deployed or updated manually without Terraform or OpeTofu, the live infrastructure no longer matches the IaC state and configuration. This often happens because developers find that the IaC path is too slow, complex, or fragile - especially when changes need to happen fast, for example, hotfixes that are then forgotten to be backported to IaC.
This can often be prevented by implementing Identity and Access Management (IAM) and Role-based access control (RBAC) concepts that prevent developers from directly applying changes within the UI, forcing them to stick to change management driven by infrastructure as code and automation. Most importantly, it means that adopting IaC requires introducing new processes and tools and a fundamental shift in an organization's culture regarding how infrastructure is managed.
2. Lack of automation and continuous deployment
Another major reason for drift is the lack of proper automation. Even when an organization adopts Terraform or OpenTofu to manage infrastructure with Infrastructure as Code (IaC), it may not automate the process effectively. As a result, developers often run relevant commands on their workstations, which can lead to errors and inconsistencies. Implementing a CI/CD (Continuous Integration/Continuous Deployment) system helps ensure that changes are made consistently, tested, and deployed automatically. This minimizes drift by streamlining the processes of code creation, testing, and deployment. Additionally, ad hoc changes may go undocumented without a clear change management strategy, further contributing to drift.
This can often be prevented by ensuring that changes can only be rolled out through automation with Pull Requests. You usually want to configure permissions in a way that disallows developers to run commands such as terraform apply , and tofu apply locally at all.
3. Automated changes outside of IaC
Another frequent reason for drift is automated changes that aren’t part of your IaC. Often, those are maintenance patches directly applied by cloud providers or third-party automation systems that directly access and modify your infrastructure resources without Terraform’s awareness, creating the perfect foundation for drift. A typical example is an auto scaler responsible for scaling the number of nodes of specific systems, such as EC2 instances. When the number of replicas scales up and down, your IaC might no longer be in sync with the deployed resources.
This can often be prevented by ignoring specific attributes in your Terraform configuration using Terraform resource lifecycles. Not all automation tools are created equal, and using third-party automation software can be a Terraform drift culprit.
4. Resource Eviction
While resource eviction also fits into the “automated changes outside of IaC” category, it’s important to address this cause for drift as it comes with its characteristics. Cost-saving measures often cause resource eviction and policy violations that can cause resources to be evicted or deleted, which can cause drift. It’s important to note that the drift caused by both is often desirable and should hence be backported to your configurations.
Here are the two primary cases:
Resource eviction through cost-saving policies, whereas automated mechanisms, will scale down your infrastructure if low traffic occurs to save costs or even partially delete entire infrastructure resources. While this core promise of elasticity is one of the main promises of the cloud and is often referred to in best practices (e.g., AWS Well-Architected Framework), it is also one of the main reasons for drift.
Resource eviction is achieved through automated policy changes, whereas automated mechanisms automatically adjust the configuration of your infrastructure if any security vulnerabilities or misconfigurations are detected. While automatically resolving those misconfigurations is desirable, it also causes our infrastructure to drift if changes aren’t monitored and backported properly.
Drift caused by resource eviction can only be addressed by backporting changes to your infrastructure. For that, it’s important to have proper monitoring practices in place to detect, understand and finally remediate drift by backporting changes to IaC.
5. Conflicting Resources
Another common reason for drift is using conflicting resources. This describes a situation in which the same cloud resource is managed with IaC in multiple places, thus causing conflicts. Those conflicts are often hard to identify and understand, especially when the related IaC is managed in multiple repositories.
This can be prevented using solutions like Terramate Cloud, which automatically detects and warns you of conflicting resource declarations, allowing you to identify and resolve them effectively.
The Impact and Risks of Drift
Now that we have learned what drift is, why it exists, and its most common reasons, let’s understand its impact and risk and why it is essential to address it. Again, infrastructure drift is untracked changes. These untracked changes pose risks of varying severity and can have a drastic impact on your infrastructure. Similarly, some changes may be beneficial for your infrastructure, improving attributes like reliability, security, and performance (remember, drift can also be caused by policies automatically correcting your infrastructure configuration in case of detected security vulnerabilities and misconfigurations).
The following sections address the risks caused by drift to your infrastructure, which can impact security, performance, and overall stability.
Security Risk
Infrastructure drift can lead to severe security risks. For instance, security group rules might be manually adjusted for testing purposes, granting unintended public access, which exposes your resources to potential attacks or unauthorized access.
Potential consequences include data breaches, financial setbacks, compliance breaches, and damage to the company’s reputation.
Compliance Risk
Drift can lead to compliance breaches, posing major risks that can significantly affect your infrastructure and your organization from a business point of view. Drift caused by automated policy adjustments or manual configuration changes can lead to breaches of regulatory requirements — for example, drift that results in users' personal data being exposed to the public or actions that enable unauthorized access to data and resources.
Performance Risk
Infrastructure drift can impair your infrastructure's performance because of unknown and unwanted changes, such as under-provisioned resources, low networking throughput, and disabling auto-scaling configurations. Drift also makes identifying, analyzing, and investigating the root cause of those issues more challenging. Unknown and untracked changes introduce challenges that increase downtime and impact resolution time.
Financial Risk
Infrastructure drift can also have financial implications. Human-induced changes might increase operational costs and lead to significantly underutilized resources. For example, imagine a developer who changes an RDS DB instance type from db.m5d.xlarge to db.m5d.4xlarge in the AWS console, resulting in 4x increased costs.
Operational Risk
If you run terraform apply or tofu apply without looking at the plan or addressing drift in the first place, Terraform and OpenTofu might make unintended changes - potentially recreating or modifying resources because of changes introduced by drift, risking downtime or data loss.
Drift can also break automated pipelines. When the live state is out of sync, tests and validation steps may behave unexpectedly, causing deployment failures or infrastructure inconsistencies.
The longer drift goes unaddressed, the more difficult it becomes to realign your infrastructure with Terraform, increasing the complexity and effort required for maintenance.
Drift in Terraform and OpenTofu: A Practical Example
Now that we understand the risks of infrastructure drift, let's look at a simple example with OpenTofu and an AWS EC2 instance to see how drift happens.
Initial OpenTofu Configuration
You start by defining an EC2 instance in your OpenTofu configuration file, e.g. main.tf :
data "aws_ami" "ubuntu" {
most_recent = true
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
owners = ["099720109477"] # Canonical
}
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = data.aws_ami.ubuntu.id
instance_type = "t2.micro"
tags = {
drift_example = "v1"
Name = "ec2-drift"
}
}Run the tofu init command to initialize your OpenTofu working environment by downloading all dependencies, such as the required providers.
$ tofu init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.74.0...
- Installed hashicorp/aws v5.74.0 (signed, key ID 0C0AF313E5FD9F80)
Providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
<https://opentofu.org/docs/cli/plugins/signing/>
OpenTofu has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that OpenTofu can guarantee to make the same selections by default when
you run "tofu init" in the future.
OpenTofu has been successfully initialized!
You may now begin working with OpenTofu. Try running "tofu plan" to see
any changes that are required for your infrastructure. All OpenTofu commands
should now work.
If you ever set or change modules or backend configuration for OpenTofu,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.Deploy the Infrastructure
After defining the configuration of your infrastructure, you run tofu apply . This command provisions the EC2 instance exactly as specified in your configuration.
$ tofu apply -auto-approve
data.aws_ami.ubuntu: Reading...
data.aws_ami.ubuntu: Read complete after 0s [id=ami-0887e1d5e322290cf]
OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
+ create
OpenTofu will perform the following actions:
# aws_instance.example will be created
+ resource "aws_instance" "example" {
+ ami = "ami-0887e1d5e322290cf"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ disable_api_stop = (known after apply)
+ disable_api_termination = (known after apply)
+ ebs_optimized = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ host_resource_group_arn = (known after apply)
+ iam_instance_profile = (known after apply)
+ id = (known after apply)
+ instance_initiated_shutdown_behavior = (known after apply)
+ instance_lifecycle = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ monitoring = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ placement_partition_number = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ spot_instance_request_id = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "ec2-drift"
+ "drift_example" = "v1"
}
+ tags_all = {
+ "Name" = "ec2-drift"
+ "drift_example" = "v1"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
aws_instance.example: Creating...
aws_instance.example: Still creating... [10s elapsed]
aws_instance.example: Creation complete after 14s [id=i-0a8cce0192cf972fe]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Introduce drift
Once the instance is live, you manually update the tag drift_example from v1 to v2 using the AWS Management Console or AWS CLI.
Run a refresh-only plan
By default, Terraform and OpenTofu compare your state file to real infrastructure whenever you invoke terraform plan , tofu plan or terraform apply , tofu apply . The refresh updates your state file in memory to reflect the actual configuration of your infrastructure. This ensures that Terraform and OpenTofu determine the correct changes to make to your resources.
If you suspect that your infrastructure configuration has drifted, you can use a -refresh-only flag to inspect what the changes to your state file would be.
Running tofu plan -refresh-only or terraform plan -refresh-only is preferred because it is safer than using the refresh commands, which automatically overwrite your state file without displaying the updates.
Run tofu plan -refresh-only to determine the drift between your current state file and the actual configuration.
$ tofu plan -refresh-only
data.aws_ami.ubuntu: Reading...
data.aws_ami.ubuntu: Read complete after 1s [id=ami-0887e1d5e322290cf]
aws_instance.example: Refreshing state... [id=i-0a8cce0192cf972fe]
^[
Note: Objects have changed outside of OpenTofu
OpenTofu detected the following changes made outside of OpenTofu since the last "tofu apply" which may have affected this
plan:
# aws_instance.example has changed
~ resource "aws_instance" "example" {
id = "i-0a8cce0192cf972fe"
~ tags = {
"Name" = "ec2-drift"
~ "drift_example" = "v1" -> "v2"
}
~ tags_all = {
~ "drift_example" = "v1" -> "v2"
# (1 unchanged element hidden)
}
# (30 unchanged attributes hidden)
# (8 unchanged blocks hidden)
}
This is a refresh-only plan, so OpenTofu will not take any actions to undo these. If you were expecting these changes then
you can apply this plan to record the updated values in the OpenTofu state without changing any remote objects.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so OpenTofu can't guarantee to take exactly these actions if you
run "tofu apply" now.As you can see by looking at the outputs, OpenTofu has detected differences between the live infrastructure and the current OpenTofu state. The refresh-only plan output indicates that OpenTofu will update your state file to modify the configuration of your EC2 instance to reflect the updated value for the drift_example tag.
You can apply these changes to make your state file match your live infrastructure but not your OpenTofu configuration with a refresh-only apply.
$ tofu apply -refresh-only
data.aws_ami.ubuntu: Reading...
data.aws_ami.ubuntu: Read complete after 0s [id=ami-0887e1d5e322290cf]
aws_instance.example: Refreshing state... [id=i-0a8cce0192cf972fe]
Note: Objects have changed outside of OpenTofu
OpenTofu detected the following changes made outside of OpenTofu since the last "tofu apply" which may have affected this
plan:
# aws_instance.example has changed
~ resource "aws_instance" "example" {
id = "i-0a8cce0192cf972fe"
~ tags = {
"Name" = "ec2-drift"
~ "drift_example" = "v1" -> "v2"
}
~ tags_all = {
~ "drift_example" = "v1" -> "v2"
# (1 unchanged element hidden)
}
# (30 unchanged attributes hidden)
# (8 unchanged blocks hidden)
}
This is a refresh-only plan, so OpenTofu will not take any actions to undo these. If you were expecting these changes then
you can apply this plan to record the updated values in the OpenTofu state without changing any remote objects.
Would you like to update the OpenTofu state to reflect these detected changes?
OpenTofu will write these changes to the state without modifying any real infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.A refresh-only operation does not attempt to modify your infrastructure to match your OpenTofu configuration. It only gives you the option to review and track the drift in your state file.
If you ran the tofu plan or tofu, apply without the -refresh-only flag now, and OpenTofu will attempt to revert your manual changes. Instead, you will update your OpenTofu configuration in the main.tf to update the tag and backport the changes. Update your code to match the following:
data "aws_ami" "ubuntu" {
most_recent = true
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
owners = ["099720109477"] # Canonical
}
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = data.aws_ami.ubuntu.id
instance_type = "t2.micro"
tags = {
drift_example = "v2" # <- updates the tag to v2
Name = "ec2-drift"
}
}Now that we have updated our configuration to match the drifted resources, we can run tofu apply , and you will see that OpenTofu doesn’t indicate any changes.
$ tofu apply
data.aws_ami.ubuntu: Reading...
data.aws_ami.ubuntu: Read complete after 0s [id=ami-0887e1d5e322290cf]
aws_instance.example: Refreshing state... [id=i-06ff4a3d0680c0913]
No changes. Your infrastructure matches the configuration.
OpenTofu has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.On the contrary, if you would have decided to not backport the changes and instead run tofu apply on the previous version of the code, OpenTofu would have indicated a change in the tag version and reconciled the drift from v2 back to v1 .
How to avoid Drift in the first place?
While it’s unrealistic to completely avoid drift, implementing some best practices helps to drastically reduce its occurrence.
In this section, we will explore the most important measurements that allow you to lower the amount of drift in your Terraform and OpenTofu projects over time.
Use a VCS to version your Code
It’s a must to keep and version your infrastructure code in a Version Control System (VCS) such as GitHub, GitLab or BitBucket. Most of the time, you would practice GitOps, meaning your main branch should be the only source of truth for your live infrastructure, and changes to the main branch are introduced via Pull Requests.
Additionally, you can implement checks that validate the integrity of your changes in Pull Requests before merging them back to the main branch by validating the code for integrity using commands such as terraform validate and tofu validate or by using additional tooling for, e.g., predicting changes when it comes to costs or for detecting security and vulnerability misconfiguration. To mention some of the most popular third-party tools available: infracost, trivy, checkov, and terrascan.
Automate Terraform and OpenTofu with CI/CD
Another must-have is to automate commands such as terraform plan , terraform apply , tofu plan and tofu apply using CI/CD. Most of the popular VCS platforms have built-in CI/CD offerings that do the job, such as GitHub Actions, GitLab CI/CD, and BitBucket Pipelines. It’s recommended that previews of changes in Pull Requests be provided to enforce the review of changes by peers and automated tooling before eventually rolling out changes when merging back to the main branch workflows, such as the apply-after-merge workflow.
Implement an IAM / RBAC Concept
As mentioned earlier in this guide, most drift is caused by manual changes made outside of Terraform and OpenTofu. To prevent those, it’s recommended to at least lock down access for teams in production environments, allowing teams to inspect deployed infrastructure using the cloud vendors’ UIs only. For that, it’s useful to design an Identity and Access Management (IAM) and Role-based Access Control (RBAC) concept that describes the different access levels in detail. It’s often common to provide developers with sandbox environments, allowing them to quickly try out infrastructure without going through infrastructure as code but then enforce any deployments to staging or production environments with infrastructure as code exclusively.
Implement a process for managing Drift
Lastly, it’s recommended that a process for managing drift is implemented. As mentioned earlier, you will be able to minimize the occurrence of drift when implementing and following some best practices, but you won’t be able to avoid it completely. That’s why it’s essential to regularly check for drift and to remediate it either by reconciling drift or by backporting changes back to your code base. If you are interested in learning how Terramate can help you detect and manage drift, please look at our guide “How to detect, manage and remediate drift in Terraform and OpenTofu with Terramate.”
How to detect and manage Drift?
Now that we have learned how to minimize drift let’s examine how to detect and manage drift for the remaining scenarios. As mentioned earlier in this guide, it’s extremely important to be aware of situations when drift occurs and remediate it effectively so that we can avoid situations such as breaking pipelines, security vulnerabilities and cost explosions.
Periodic Drift Detection Checks
Regular drift detection ensures your infrastructure stays in sync with Terraform. Running terraform plan and tofu plan as scheduled jobs in, e.g., GitHub Actions periodically is an easy way to catch any discrepancies between your desired and actual state.
Automating drift checks in your CI/CD pipeline helps spot misalignments early while scheduling terraform refresh and tofu refresh keeps your state file in step with real-time changes.
You can also use tools like AWS CloudTrail or Google Cloud Audit Logs to track manual changes outside Terraform. These logs enable you to detect drift early, allowing for a prompt response before it becomes an issue.
Visibility and Notification
To guarantee the security and compliance of your infrastructure, you must have a reliable visibility and alerting for drift in place that your team can use to monitor drift. For example, Terramate Cloud comes with a concept of Alerts and Slack Notifications that help you manage and observe drift. At the bare minimum, you want to have at least a Slack channel that notifies all relevant teams about newly detected drift. Just running periodic checks is not enough and often not even noticed by teams, as it’s not actionable.
Remediating Drift in Terraform and OpenTofu
The final step for managing drift in Terraform and OpenTofu is called remediation, whereas two methods exist:
- Reconciliation
The goal of reconciliation is to return everything to the desired configuration in your Terraform or OpenTofu configurations. This is the go-to approach when changes made outside your IaC need to be reversed and can be done by running terraform apply or tofu apply again.
- Aligning the code/backporting changes
In case important changes have been done outside your IaC by e.g., policy checks to resolve a security vulnerability in your configuration, you must backport this change back to your infrastructure code. This can be done by adjusting your configuration accordingly. To validate that all changes have been backported, you can run terraform plan or tofu plan , which should result in an empty plan that simply does not indicate any changes.
How to detect, manage and remediate drift in Terraform and OpenTofu with Terramate
Drift detection and remediation is hard, and neither Terraform Community Edition nor OpenTofu CLI provides additional tools for detecting and managing drift properly. That’s why we have added developer-centric features such as drift detection observability, alerts and notifications in Terramate Cloud, which help you bring visibility and tools to manage drift effectively. If you are interested in learning how Terramate can help you bring your drift to a bare minimum and to manage it effectively, we recommend you take a look at our guide “How to detect, manage and remediate drift in Terraform, OpenTofu and Terragrunt with Terramate”.
Congratulations if you stayed with us until now. You just learned a bunch about drift in Terraform and OpenTofu. If you have any questions about this topic or feedback for us in general, we encourage you to join our Discord Community to get in touch with our team.