How to keep your Terraform code DRY by using Terramate

Introduction

Terramate is a tool for managing multiple Terraform stacks that comes with support for change detection and code generation. We invented Terramate to help engineering teams to become more productive when working with Terraform. We invented Terramate to help people like you.

Terramate invents the concept of stacks which are smaller and isolated units of Terraform code and states that help you to split huge code bases into smaller executable units.

If you want to learn why we invented Terramate, please see our introduction blog article.

In this blog post, we will elaborate on Terramate globals, an advanced concept in Terramate that will help you to share data among stacks and to keep your code DRY.

What is DRY and why is it important?

DRY stands for “Don’t Repeat Yourself”. It is a software engineering principle that aims to reduce software complexity. Any piece of information that is duplicated is liable to get stale and cause problems and requires extra work to be kept updated, increasing the costs and difficulty of maintaining software.

In order to avoid paying the extra costs of maintaining duplicated information, we need abstractions that help us:

• extract information, define it only once, and reuse it whenever possible.
• provide an easy way to update and access the single representation of information and use it multiple times for different environments.

In order to provide such abstraction, Terramate supports sharing data globally using Terramate globals.

How do Terramate Globals help?

Globals provides a way to define reusable and inherited data across stacks in a clear hierarchical manner whilst providing a simple merge semantic.

Creating globals is done using the globals block in a way that is very similar to Terraform locals :

globals {
  env = "staging"
}

How to define globals in Terramate

Terramate Globals can be defined on any Terramate configuration file in any directory inside a Terramate project. Globals are always lazily evaluated in the context of a Terramate stack. What makes Terramate Globals different and a great way to deal with duplication is its hierarchical behavior.

Globals defined on a directory will be available to all stacks that are children of this directory.

Summarizing Terramate Globals behavior:

• A global variable is defined in the globals block within any Terramate configuration file.
• You can have multiple global blocks that each can define zero to multiple global variables.
• A global variable can not be redefined at the same hierarchical level.
• A global variable can be redefined on different hierarchical levels while lower levels (e.g stack level) override higher levels (e.g. root level or parent level).
• A global variable can reference other global variables and Terramate metadata as long as no cycles are created. Also Terramate Functions (tm_<function>() ) are available for use within globals.
• Each global variable is lazily evaluated on the stack level.

Hint: If unsure, global variables follow the same rules as you know from Terraform local variables.

But let’s dive deep by transforming a normal Terraform setup with a lot of code duplication and maintainability overhead into a lean Terramate setup using code generation.

Pure Terraform Configuration

A Terraform Infrastructure as Code (IaC) setup with multiple stacks that deploy infrastructure on Google Cloud might be organized like this:

Each stack defines resources in main.tf , defines a provider configuration provider.tf , and a Terraform backend configuration in backend.tf .

projects/
├── my-project-prod/
│   ├── stack-1/
│   │   ├── main.tf
│   │   ├── provider.tf
│   │   └── backend.tf
│   └── stack-2/
│   │   ├── main.tf
│   │   ├── provider.tf
│   │   └── backend.tf
└── my-project-staging/
    ├── stack-1/
    │   ├── main.tf
    │   ├── provider.tf
    │   └── backend.tf
    └── stack-2/
        ├── main.tf
        ├── provider.tf
        └── backend.tf

Example project structure for a Google Cloud Terraform project

But then on each stack, we see the same duplicated provider configuration in provider.tf just defining different project setting for the provider depending on the environment of staging or prod :

# File: /projects/my-project-<env>/<stack>/provider.tf

provider "google" {
  project = "my-project-<env>"
  region  = "europe-north1"
}

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.0"
    }
  }
}

terraform {
  required_version = "1.2.3"
} 

Example Terraform provider.tf

This configuration is increasingly annoying to maintain in a growing IaC setup in at least the following situations:

• Updating the provider or Terraform version constraints in all stacks is a manual effort.
• Promoting a stack from staging to production requires manually updating the project setting of the provider.

Terramate Configuration

Globals and code generation can help teams define the settings once and generate the file depending on the stack’s position in the hierarchy.

This is done through Terramate configuration files. Any HCL file with the following suffixes:

• .tm
• .tm.hcl

Will be recognized as a Terramate configuration.

Globals, Metadata, and Functions

Before we dive into the configuration let us shortly check what features are available.

• Terramate Globals — This is what this blog post is about and you already got some details on what global variables are and will soon get an idea on how to make use of them.
• Terramate Meta Data — This is information provided through the terramate namespace. It includes information regarding the stack (e.g. the name or the path) and also project-wide information that then can be used on code generation and within global variables. For details please see the docs on metadata.
• Terramate Functions — All functions provided by Terraform v0.15.3 are provided by Terramate prefixed with tm_ . In code generation, you can use tm_length() when you want to calculate a length on generation time, or use length() if you want to generate the code leaving the Terraform function call in the generated code.

Generate Terraform Stacks with Terramate

The first step towards a Terramate configuration is to mark all stack directories as stacks by creating a file in each stack containing a stack {} block e.g. a file called stack.tm.hcl .

# File: /projects/<project>/<stack>/stack.tm.hcl
stack {
  name        = "My awesome <stack>"
  description = "This stack defines awesome resources"
}

How to define Terramate stacks in stack.tm.hcl files

Generate Terraform Provider definitions with Terramate

By creating the following providers.tm.hcl file on a parent directory of all stacks we can simplify maintainability and enable code generation.

# File: /projects/providers.tm.hcl

globals {
  # The default project_id to configure the provider with.
  # The provider_id can be set at any level in the hierarchy.
  # If no project_id the provider fallbacks are used.
  terraform_google_provider_project = tm_try(global.project_id, null)

  # The default region to configure
  terraform_google_provider_region = "europe-north1"

  # Version constraints (can be overridden at lower levels, e.g. stack level)
  terraform_google_provider_version = "4.0"
  terraform_version                 = "1.2.3"
}

# Create a provider config in every stack reachable from this config.
# The file is prefixed with _terramate_generated to make the generated nature
# of it more visible
generate_hcl "_terramate_generated_providers.tf" {
  content {
    # The provider configuration
    provider "google" {
      project = global.terraform_google_provider_project
      region  = global.terraform_google_provider_region
    }

    # Provider version constraints
    terraform {
      required_providers {
        google = {
          source  = "hashicorp/google"
          version = global.terraform_google_provider_version
        }
      }
    }

    # Terraform version constraints
    terraform {
      required_version = global.terraform_version
    }
  }
}

Terraform provider definition in providers.hcl.tm

To define the differences in the staging and prod environment project directories we also create those two files and define globals that will be inherited in each environment's stacks.

# File: /projects/my-project-staging/project.tm.hcl
globals {
  env        = "staging"
  project_id = "my-project-${global.env}"
}

Defining Terramate globals for staging

# File: /projects/my-project-prod/project.tm.hcl
globals {
  env        = "prod"
  project_id = "my-project-${global.env}"
}

Defining Terramate globals for prod

Generate Terraform backend definitions with Terramate

Terraform’s backend configuration has even more differences in each stack as we need to define a unique prefix for the Terraform state.

Assuming, we use a bucket to store the state that follows the tf-state-<project> naming convention and that the prefix used inside the bucket should match the actual path of the stack inside of our hierarchy, we can use our already existing configuration and special Terramate metadata about each stack.

# File: /projects/backend.tm.hcl

# The file is prefixed with _terramate_generated here to make the generated nature
# of it more visible
generate_hcl "_terramate_generated_backend.tf" {
  content {
    terraform {
      backend "gcs" {
				bucket = "tf-state-${global.project_id}"
		    prefix = terramate.stack.path.absolute
      }
    }
  }
}

Generate Terraform backend.tf files with Terramate

Generating the files

As we are about to replace backend.tf and provider.tf in all stacks, we first need to delete those files manually.

To generate the new files, we need to run terramate generate once. When the configuration is changed, terramate generate needs to be called again.

Our final directory tree will look something like this now:

projects/
├── provider.tm.hcl
├── backend.tm.hcl
├── my-project-prod/
│   ├── project.tm.hcl
│   ├── stack-1/
│   │   ├── main.tf
│   │   ├── stack.tm.hcl
│   │   ├── _terramate_generated_provider.tf
│   │   └── _terramate_generated_backend.tf
│   └── stack-2/
│   │   ├── main.tf
│   │   ├── stack.tm.hcl
│   │   ├── _terramate_generated_provider.tf
│   │   └── _terramate_generated_backend.tf
└── my-project-staging/
    ├── project.tm.hcl
    ├── stack-1/
    │   ├── main.tf
    │   ├── stack.tm.hcl
    │   ├── _terramate_generated_provider.tf
    │   └── _terramate_generated_backend.tf
    └── stack-2/
        ├── main.tf
        ├── stack.tm.hcl
        ├── _terramate_generated_provider.tf
        └── _terramate_generated_backend.tf

Project structure with files generated by Terramate

Terramate can help orchestrate the execution of stacks but will not impose that, we can keep using whatever workflow we were using before. As far as Terraform planning/applying goes our project still is as it was before.

Making Terramate Globals available as Terraform Locals

If you want to make Terramate Globals available to Terraform code you can generate a file defining a locals block containing the variables you like to “export” to Terraform.

# File: /projects/locals.tm.hcl

# Export env and project_id to be referenced as local.project_id and local.env
# in main.tf and other Terraform files in each stack.
generate_hcl "_terramate_generated_locals" {
  content {
    locals {
      env        = global.env
      project_id = global.project_id
    }
  }
}

Generate Terraform locals with Terramate

When DRY is too DRY

As usual in software engineering, everything is about trade-offs and the same applies to the DRY principle. Creating a single source of truth in some scenarios will also create a central place where any changes can amplify across multiple different places.

For example, when using local Terraform modules any changes on this local module will trigger changes on any other module referencing it. It could trigger changes both in staging and a production environment, leading to a big blast radius.

But also for Terramate generated code, there will be always a trade-off on how much code you want to generate from the same source of truth for different environments. The hierarchical approach of Terramate Globals and also features like conditional code generation that is already available in Terramate can help create a more complex but still maintainable configuration of a multi-stack Terraform setup.

Summary of what Terramate Globals enable

• Updating Terraform version — Just edit the /projects/providers.tm.hcl file set a new terraform_version , and run terramate generate to update all stacks.
• Terramate also allows to update Terraform version in staging for example by redefining terraform_version in /projects/my-project-staging/project.tm.hcl first and later promoting it to all stacks.
• Promoting a stack through the environment from staging to prod by copying the full directory of the stack and running terramate generate to update and adjust the generated files.
• Make use of Terramate Globals by exporting them as Terraform locals and make use of them in your code (e.g. main.tf ) to improve the promotion of stacks through the environment.
• In general Globals and Code Generation can be already used for very complex scenarios and are a powerful way to solve some Terraform challenges.
• In addition to generating HCL files, Terramate supports generating any plain text file type using generate_file blocks.
• You do not need to adopt Terramate as an Orchestrator in order to make use of its code generation features — but anyway we encourage you to try out the change detection features of Terramate as an Orchestrator.
• In order to adapt the code generation using Terramate, not all stacks need to be included. The resulting generated code is 100% Terraform and can be checked in to git like all the rest of the code.

If you would like to learn how Terramate please find the documenation in https://terramate.io/docs/cli/.

If you are interested in learning how Terramate can help optimzing your Terraform environments or if you are interested in working with us, feel free to drop us a line at hello@terramate.io or join our Community on Discord.