Why we built Terramate

It’s late 2020, and my co-founder Marius and I just finished building an Internal Developer Platform (IDP) for one of Europe's fastest-growing consumer startups. Back then, Marius and I ran a boutique agency that specialized in building developer platforms on major clouds, helping some of Europe’s fastest-growing scale-ups design and implement platforms to sustain hypergrowth.

To give you an idea of the scale, we started working with the customer when their engineering team consisted of just five people and wrapped things up when their engineering team expanded to more than 250 people.

The primary goal of that IDP was to allow developers to deploy and manage production-grade infrastructure in a self-service manner. The customers’ CTO understood early on that building a DevOps team that grew linearly to the developer team to sustain the team's growth wasn’t a viable option and a drag on operational leverage. Thus, he decided to go all-in on platform engineering, developer experience, and self-service.

Fast-forward 18 months, we delivered a fully-fledged IDP, mainly powered by Infrastructure-as-Code (IaC) tools like Terraform and Kubernetes. Our north star metric, “How long does it take to create and deploy a new service to production”, which previously took a couple of days and depended on multiple stakeholders, was now reduced to a few minutes and done in a self-service manner.

The developers loved it, and management celebrated the reduced time to market for new services!

When we finally handed over the project, I looked back at all the great work we had done and thought: “We succeeded in such incredible ways… and we shouldn’t have needed to do any of it.”

Frankly, we were frustrated by how much time and effort required to build a platform on top of IaC, and that’s when the idea of Terramate originated.

State of the IaC Market

IaC tools such as Terraform and OpenTofu are robust and mature tools for managing infrastructure with code. Still, they have massive gaps that prevent them from being usable for teams, leading organizations to invest significant resources to overcome those:

• Lack of standards for managing multiple complex environments (e.g., in Terraform and OpenTofu, multiple approaches such as workspaces and directories exists and there’s no real standard the community agrees on)
• Limited capabilities to reduce code duplication (e.g., in Terraform and OpenTofu, it’s hard to keep code DRY because each stack requires its own provider and backend configuration)
• No built-in automation (teams need to design and configure pipeline configurations for GitOps workflows over and over again)
• Limited orchestration capabilities (e.g., in Terraform and OpenTofu, it’s impossible to define dependencies among root modules, to orchestrate those in the right order or to only orchestrate modules that contain changes)
• Lack of observability capabilities (e.g., no easy way of monitoring changes introduced in PRs and deployments, lack of DORA metrics, etc.)
• Poor visibility (especially when managing IaC multiple repositories, it’s often impossible to understand what is managed where and by whom)
• Lack of governance and policies (can be addressed by adopting additional tooling such as Open Policy Agent (OPA))
• Lack of misconfiguration detection and cost protection (can be addressed by adopting additional toolings such as trivy, checkov, infracost and others)
• Lack of self-service (Developers still need to learn how to write HCL to use Terraform and understand how infrastructure needs to be configured to deploy and manage production-grade infrastructure)

We built Terramate to eliminate the need for teams to close those gaps over and over again, stemming the tide of time and money wasted - a waste we see repeatedly happen throughout the industry today.

Why does it matter?

Today, teams are overcoming those gaps by

(a) either building complex and time-consuming home-grown solutions that often don’t scale or become maintenance rabbit holes or

(b) buying purpose-built CI/CD platforms (often referred to as TACOS - Terraform Automation and Collaboration Software) that require a significant migration investment, inflict lock-ins, and lack the flexibility to allow organizations to scale fast - while introducing new security concerns.

A quick back-of-the-envelope calculation: Imagine a DevOps team of five building and maintaining a home-grown IaC solution. If they dedicate 50% of their time to that effort, with a median salary of $150,000 per FTE, your organization will spend at least $375,000 yearly in development and maintenance costs. This ignores the opportunity costs, which often will be reflected as new headcount requests in next year's budget planning cycle.

What is Terramate, and who is it for?

Terramate is an IaC orchestration, observability and visibility platform built on years of experience working and scaling IaC for some of the world's most innovative companies. It enables DevOps, SRE and Platform teams to bridge those gaps, leading to a rapidly accelerated & de-risked IaC practice.

What can teams achieve with Terramate?

Terramate has been designed to add immediate value to any existing Terraform, OpenTofu or Terragrunt project and is the perfect starting point for any new project to ensure scalability and consistency. It helps teams save thousands of hours by building on top of a suite of capabilities that can be adapted entirely or selectively. Instead of reinventing the wheel, they can refocus on their core business priorities.

Here are some of the benefits measured by organizations that have adopted Terramate in production:

• Up to 10 times faster deployments (leading to fewer blocking pipelines, a more rapid development cycle, and a lot fewer engineers twiddling their thumbs waiting)
• Up to 50% improvement in DevOps velocity (shifting time from handling complexity to simply delivering value)
• Risk mitigation (proper governance and small blast radii enable quick anomaly detection and remediation, leading to minimal service impairments and a greatly reduced MTTR)

How Terramate is different

As mentioned earlier in this article, organizations currently follow two main approaches when adopting Infrastructure as Code:

Building and maintaining DIY solutions

This approach describes organizations that invest significant time and resources in building and maintaining DIY solutions, often by gluing various open-source tools into platforms maintained by at least 2-3 internal DevOps and Platform engineers. The outcome is platforms that often don’t scale and require constant maintenance effort, which is initially frequently underestimated. Common symptoms of DIY approaches include script sprawl, lack of conventions, disorganized project structures, unsafe CI/CD practices, and hardcoded credentials, but the real kicker is when the team of maintainers moves on to work for a different company, which usually leads to an abandoned platform that no one understands.

Buying a purpose-built CI/CD platform (TACOS) for IaC

Two different types of purpose-built CI/CD platforms exist:

1. Vendor-specific offerings such as HashiCorp Cloud Platform and Pulumi Cloud that are focused on making their open-source offering usable for teams.
2. Vendor-agnostic platforms such as Env0, Scalr, and Spacelift provide more comprehensive offerings by enabling multi-IaC use cases. All those products are purpose-built CI/CD systems that help teams automate, collaborate and govern IaC at scale.

Although existing purpose-built CI/CD platforms address some of the mentioned issues, they come with significant trade-offs:

• Require teams to structure IaC and manage environments in specific ways, often leading to large migration efforts, less flexibility, degraded agility and lock-ins
• Require full access to cloud account and state files, making them difficult to evaluate for security teams
• Only mitigate some of the gaps and are mainly focussed on CI/CD, forcing teams to buy additional vendors to close gaps in, e.g., visibility and observability

Last but not least, teams should answer the bigger question of whether buying a dedicated CI/CD platform for automating IaC is really required. While CI/CD for Infrastructure as Code was an issue a while ago, general-purpose CI/CD platforms such as GitHub Actions or GitLab CI/CD became increasingly mature over the past couple of years, leading teams to question whether those platforms provide a good ROI. This is why we have shifted Terramate’s orchestration capabilities to the client side, enabling teams to re-use their existing CI/CD systems - a simpler, more secure, and more cost-efficient approach than buying a purpose-built platform.

A new approach: The Terramate Answer

Now that we understand existing approaches to adopting and scaling IaC, let’s discuss how Terramate differentiates. Please find the following table to understand the differences between Terramate and TACOS (purpose-built CI/CD platforms), such as Hashicorp Cloud Platform (HCP) Terraform:

As you can see, Terramate differentiates by providing a platform that closes the gaps in existing IaC projects. It allows teams to re-use existing CI/CD platforms, doesn’t require access to cloud accounts or state backends and can be onboarded instantly without causing migration effort, making it highly suitable to be used in any existing project that has grown over time.

How does Terramate work?

The Terramate platform mainly consists of two components:

Terramate CLI is an open-source IaC orchestrator and code generator. It’s designed to make individual DevOps, Platform-, and Site Reliability Engineers more productive, consistent, and reliable by allowing them to:

1. Streamline environments and configuration management: Terramate CLI enables teams to group related infrastructure resources required for managing environments and services as stacks that can be provisioned and managed as a unit. It also adds code generation to keep the configuration DRY. This allows teams to efficiently provision and manage multiple services and environments and avoid manual copy-and-paste of duplicate configuration files.
2. Automate and orchestrate IaC in any CI/CD: Terramate CLI enables teams to turn CI/CD platforms such as GitHub Actions, GitLab CI/CD, BitBucket Pipeline, Azure DevOps, Jenkins and others into powerful infrastructure vending machines by providing Pull Request GitOps automation workflows with previews and checks. The git-based change detection allows only orchestration stacks containing changes in the current CI/CD run, thus lowering the blast radius to a bare minimum. This means no more wasted time and money maintaining DIY solutions or buying expensive purpose-built CI/CD systems.

Terramate Cloud is an IaC management platform. It’s a fully managed cloud service that has been designed to enable teams to collaborate more efficiently and to deliver more value to developers by allowing them to:

1. Better collaboration, governance, observability and drift control: Boost reliability and productivity with previews in PRs, policies, observability, notifications, SlackOps, and automated drift detection and reconciliation, leading to fewer failures and more stable infrastructure.
2. Get complete visibility and insights: Remove the guesswork by analyzing all infrastructure managed across all environments, repositories and teams in a centralized place. Detect and remediate events such as failed deployments and drift. Get insights into Pull Requests and deployments. Use audit logs to understand changes over time.
3. Provide IaC self-service: Pave the golden path by providing a UI to scaffold new infrastructure without having to write a single line of code, GitOps workflows, centralized configuration management and more.

Terramate Cloud works by receiving data from Terramate CLI via a push model. Terramate CLI runs on the client side, locally or in an existing, general-purpose CI/CD platform such as GitHub Actions, GitLab CI/CD, Bitbucket Pipelines, Azure DevOps and others and integrates with existing tooling such as Slack or Microsoft Teams.

Why does Terramate stand out?

Here’s an overview of design decisions and approaches that make Terramate stand out:

• Graph-based orchestration: Terramate shifts orchestration capabilities to the client side. The orchestration engine in Terramate CLI allows teams to efficiently orchestrate and automate your IaC locally or in any CI/CD, essentially adding capabilities of purpose-built CI/CD platforms for IaC to any general-purpose CI/CD platform such as GitHub Actions, GitLab CI/CD, BitBucket Pipelines, Azure DevOps and others.
• Unlimited concurrency and up to 90% faster pipelines: Terramate enables unlimited concurrent workflows that run in any CI/CD. With change detection and parallel execution, you can run pipelines up to 90% faster and significantly lower your build minute consumption. Say goodbye to sequential and blocking pipelines.
• Native IaC without lock-In: With Terramate, you don’t need to adopt another syntax to get value. If you use Terramate to reduce code duplication with code generation, it will always generate native IaC, meaning you don’t shift away from native IaC.
• Full flexibility: Terramate allows you to manage environments your way. Workspaces, tfvars, Partial Backend, Directories, Terragrunt - Terramate supports any approach and doesn’t inflict any limitations upon teams.
• Unmatched security: Terramate doesn't require access to your code, state, or cloud account. It works by extracting data such as resources and meta-data from plans on the client side, sanitizing it and syncing it from Terramate CLI to Terramate Cloud using a push model, making it the most secure approach in the market.
• Minimal blast radius: Terrmate helps teams split up large state files into smaller stacks and minimize the blast radius risk of your IaC stacks to a bare minimum by executing stacks containing changes only.
• Fastest time-to-value: Terramate can be onboarded to any Terraform, OpenTofu or Terragrunt project within minutes, providing immediate value! No refactoring is required!

Summary

In this article, we explained why we built Terramate, who benefits from using it, and how it differentiates from other approaches in the market. If you want to learn more about how Terramate can help your organization supercharge your IaC projects, please book a demo with one of our solutions architects.

If you prefer to explore Terramate on your own, please take a look at the getting started section in our documentation.