A pattern for Terraform stacks

I want to outline a pattern I have been working on for Terraform stacks.

To understand what I am talking about in the rest of this blog post you must be aware of the following concepts:

1. A Terraform module is a configuration of one or more resources. These are the building blocks from which you build larger cloud architectures. They are the foundation for Terraform stacks, just like they have always been the foundation of everything in Terraform.
2. A stack component is an instance of a Terraform module that is part of a Terraform stack. A component always use one Terraform module as its source. You use a component block to configure a component, and you configure them in files ending in .tfcomponent.hcl.
3. You can group multiple stack components into logical constructs called stack component configurations. These represent a partial Terraform stack in the sense that they include a number of component blocks in .tfcomponent.hcl files but not any .tfdeploy.hcl files. You publish them to your private module registry on HCP Terraform similar to how you publish regular Terraform modules. You reference these stack component configurations using stack blocks within another stack configuration:

   stack "my_stack" {
     source  = "app.terraform.io/my-organization/my-stack-component-configuration"
     version = "1.0.0"
   
     inputs = {
        # ...
     }
   }

An interlude on Terraform module design

#

A few design principles for Terraform modules I have come to adopt are:

• Parametrize everything. You will always end up regretting it if you don’t. What this means is that you should be able to configure everything of interest for a given resource through the variables of your module.
• If your module creates multiple copies of something, use a map() input variable together with for_each. Don’t try to be clever and construct a map within the module from an input that is not a map.
• Don’t do “1 module = 1 resource”, except for where it makes sense. At the same time, most modules should likely not contain more than a handful of resources and only where it makes sense (e.g. an Azure VNet module).

Apart from that I use a lot of common sense when I write my Terraform module code. It is not that complicated to write the code and of course anyone has access to an LLM that can do it for you these days. The difficult part is always in getting the abstractions right. There is simply no correct answer to that problem, it will always depend on your context.

An important detail for the rest of this blog post is that since I parametrize everything these modules tend to be very general in the sense that I could use them to configure my abstractions in many different ways. A drawback is that you usually end up with many variables, so make sure to provider sensible defaults for most of them.

Stacks within stacks within stacks …

#

Imagine you have a number of Terraform modules for different abstractions for HCP Terraform resources. With these modules you can put together a “HCP Terraform landing zone”¹ that is the foundation for a team to get started provisioning their own cloud infrastructure through HCP Terraform. Whenever I mention landing zone in the rest of this blog post I will mean these types of landing zones, nothing else.

A landing zone will look more or less the same for developer teams targeting Azure, AWS, or any other cloud environment. To make my life easier I will assume the developers I talk about either work with Azure or AWS.

You could build a generalized stack component configuration with the required components (thus, Terraform modules) that these types of landing zones require. This could be modules for project, workspaces, variable sets, policy set assignments, teams, etc.

However, some things might be different between an Azure landing zone and an AWS landing zone². At the very least the names of the environment variables you set for workload identity federation authentication to Azure and AWS are different. If your stack component configuration are built from general Terraform modules you should be able to build a landing zone targeting any cloud, but you could also take the abstraction one more step.

This leads us to building two new stack component configurations:

1. One for an Azure landing zone, and
2. One for an AWS landing zone.

These new stack component configurations work as wrappers for the generalized stack component configuration discussed above, but it makes the necessary changes to adapt it for the target cloud.

What kind of adaptions might this be? A few examples I can think of are:

• Preconfigure names of variables used for OIDC workload identity federation authentication.
• Add additional components (e.g. for Azure landing zones we want to apply additional policy sets that are not part of the generalized stack component configuration).
• Add provider-specific input variables or output values.

In this way you can keep the generalized stack component configuration as it is, and add provider-specific adjustments in its own abstraction layer. The consuming Terraform stack will not need to know these details, they will just have to provide the required input values.

What does it look like in practice? I will just sketch out the blocks without the content.

The initial stack component configuration (the base landing zone) contains a number of component blocks:

# variables.tfcomponent.hcl
variable "one" { ... }
variable "two" { ... }
variable "three" { ... }

# providers.tfcomponent.hcl
provider "tfe" "platform" { ... }

# components.tfcomponent.hcl
component "first" {}
component "second" {}
component "third" {}

# outputs.tfcomponent.hcl
output "uno" { ... }
output "dos" { ... }
output "tres" { ... }

As you can imagine it might include additional blocks of each kind.

This stack component configuration is published as hcp-terraform-landing-zone-base in your private registry on HCP Terraform.

The Azure abstraction layer starts by referencing the hcp-terraform-landing-zone-base stack component configuration in a stack block:

# components.tfcomponent.hcl

stack "default" {
  source  = "app.terraform.io/my-organization/hcp-terraform-landing-zone-base"
  version = "1.0.0"

  inputs = { ... }
}

The Azure abstraction will likely include most of the variable blocks as in the base layer, but some of these variables might be hard-coded according to the requirements of the Azure adjustments (I will let you figure out what these might be in your context, but typically it could be names of things or tag keys and values).

There could also be additional variable blocks that are not present in the base layer. You could also add additional component blocks if needed. This Azure landing zone is published as its own stack component configuration named hcp-terraform-landing-zone-azure to the private registry.

Now, in your actual Terraform stack where you provision HCP Terraform landing zones for Azure workloads you reference your Azure abstraction:

# my-repo/stacks/azure-landing-zones/components.tfcomponent.hcl

stack "hcp_terraform_landing_zone" {
  source  = "app.terraform.io/my-organization/hcp-terraform-landing-zone-azure"
  version = "1.0.0"

  inputs = { ... }
}

The image above can be extended to visualize this:

In this layer you will likely have additional stack blocks and component blocks making up the larger infrastructure you are provisioning. There could also be additional variable blocks. Of course this is also where you will find one or more deployment blocks in accompanying .tfdeploy.hcl files.

Be aware that there might be a lot of variables to pass through these layers. Hopefully you can abstract some of these away in the middle layer, but this is not always possible.

… within stacks …

#

If you are wondering: yes, you could add more layers in this Terraform stack cake. Should you? If you have a use case where this makes sense I would say yes! However, for my needs I have found that one extra level of abstraction is sufficient.