Terraform Stacks with Microsoft Azure

Are you curious about Terraform stacks and want to get started with stacks for Microsoft Azure, then this guide is for you.

An Introduction to Terraform Stacks

#

A stack consists of one or more components. Each component is created in one or more instances called deployments. A component is similar to a Terraform module, and a deployment is similar to an instance of a Terraform module with a given set of input values.

Examples of possible components in a Terraform stack on Azure are:

• A resource group component
• A virtual network component
• A storage account component
• A function app component

You build your Terraform stack from a number of components. Next you create deployments of these components. One deployment creates one instance of each component that is part of the stack.

Reasons for adding different deployments on Azure include:

• Create multiple different environments (development, staging, production)
• Create copies of your infrastructure in multiple different locations (Sweden Central, West Europe, East US)
• Create copies of your infrastructure in multiple different Azure subscriptions

I think a more logical way of visualizing a stack with its deployments and components, is this:

In this post we will consider two components:

• A resource group component
• A storage account component

These components will be created in three deployments:

• development
• staging
• production

Apart from components and deployments, there is one more concept to introduce: orchestration rules. An orchestration rule allows us to specify conditions for when a plan operation for a deployment should be automatically approved. The orchestration rule has access to a context variable with results from the plan phase.

Dynamic Credentials with Azure

#

A prerequisite to work with stacks is to be able to authenticate to the target provider.

Stacks use workload identity for provider authentication. This is a more secure way of interacting with Azure from HCP Terraform. However, there is a setup step where you configure a trust relationship between HCP Terraform and Azure.

Once this trust relationship is set up the interaction between HCP Terraform and the target platform (Azure in this case) follows this pattern:

The trust relationship is configured for each HCP Terraform organization, project, stack, deployment, and operation (either plan or apply).

Create a new Terraform configuration (i.e. an empty directory).

Create a file named variables.tf with variables for your stacks deployment names, your HCP Terraform organization name, your HCP Terraform project name, and finally the stack name:

variable "deployment_names" {
  type        = list(string)
  description = "List of Terraform stack deployment names"
}

variable "organization_name" {
  type        = string
  description = "HCP Terraform organization name"
}

variable "project_name" {
  type        = string
  description = "HCP Terraform project name"
}

variable "stack_name" {
  type        = string
  description = "Terraform stack name"
}

Create a variables file named terraform.tfvars with the following content (change the placeholders for your values):

deployment_names  = ["development", "staging", "production"]
organization_name = "<Your HCP Terraform organization name>"
project_name      = "<Your HCP Terraform project name>"
stack_name        = "azure-stack"

Create a file named providers.tf, configure both the AzureRM provider and the Azure AD (i.e. Entra ID) provider:

terraform {
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.0.2"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

provider "azuread" {}

provider "azurerm" {
  features {}
}

I am using implicit provider configurations coming from my Azure CLI. If you don’t have the Azure CLI installed I recommend that you do it. Instructions are available in the documentation.

Create a file named main.tf and add the following configuration:

# data about the current subscription
data "azurerm_subscription" "current" {}

# create an app registration
resource "azuread_application" "hcp_terraform" {
  display_name = "hcp-terraform-azure"
}

# create a service principal for the app
resource "azuread_service_principal" "hcp_terraform" {
  client_id = azuread_application.hcp_terraform.client_id
}

# assign the contributor role for the service principal
resource "azurerm_role_assignment" "contributor" {
  scope                = data.azurerm_subscription.current.id
  principal_id         = azuread_service_principal.hcp_terraform.object_id
  role_definition_name = "Contributor"
}

# create federated identity credentials for **plan** operations
# for each deployment name
resource "azuread_application_federated_identity_credential" "plan" {
  for_each       = toset(var.deployment_names)
  application_id = azuread_application.hcp_terraform.id
  display_name   = "stack-deployment-${each.value}-plan"
  audiences      = ["api://AzureADTokenExchange"]
  issuer         = "https://app.terraform.io"
  description    = "Plan operation for deployment '${each.value}'"
  subject = join(":", [
    "organization",
    var.organization_name,
    "project",
    var.project_name,
    "stack",
    var.stack_name,
    "deployment",
    each.value,
    "operation",
    "plan"
  ])
}

# create federated identity credentials for **apply** operations
# for each deployment name
resource "azuread_application_federated_identity_credential" "apply" {
  for_each       = toset(var.deployment_names)
  application_id = azuread_application.hcp_terraform.id
  display_name   = "stack-deployment-${each.value}-apply"
  audiences      = ["api://AzureADTokenExchange"]
  issuer         = "https://app.terraform.io"
  description    = "Apply operation for deployment '${each.value}'"
  subject = join(":", [
    "organization",
    var.organization_name,
    "project",
    var.project_name,
    "stack",
    var.stack_name,
    "deployment",
    each.value,
    "operation",
    "apply"
  ])
}

I have added federated credentials for each deployment in the stack, and for each deployment I add both plan and apply credentials. You could create separate service principals for each of the credentials if you want to limit what RBAC permissions each has. However, for this demo I decided to use a single service principal on Azure. Note that there is a limit for 20 federated identity credentials for a given application/service principal.

Finally, create a file named outputs.tf with the following content:

output "configuration" {
  value = {
    client_id       = azuread_service_principal.hcp_terraform.client_id
    tenant_id       = data.azurerm_subscription.current.tenant_id
    subscription_id = data.azurerm_subscription.current.subscription_id
  }
}

You will need these output values later when we configure our stack in HCP Terraform.

Initialize the Terraform configuration:

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/azuread versions matching "~> 3.0.2"...
- Finding hashicorp/azurerm versions matching "~> 4.0"...
- Installing hashicorp/azuread v3.0.2...
- Installed hashicorp/azuread v3.0.2 (signed by HashiCorp)
- Installing hashicorp/azurerm v4.4.0...
- Installed hashicorp/azurerm v4.4.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Next, apply the configuration:

$ terraform apply -auto-approve
...
Plan: 9 to add, 0 to change, 0 to destroy.
...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

configuration = {
  "client_id" = "<your client id>"
  "subscription_id" = "<your subscription id>"
  "tenant_id" = "<your tenant id>"
}

Looking at the credentials for the app registration in our Azure tenant we can see that all the credentials have been created.

We are now ready to work with Terraform stacks.

Terraform Stacks CLI

#

Terraform stacks come with a dedicated CLI tool named tfstack.

There are a few different options for how to install it. You can find the binary for different architectures on the HashiCorp releases website.

I am using a MacBook, so I prefer to use Homebrew:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/tfstacks
==> Auto-updating Homebrew...
==> Fetching hashicorp/tap/tfstacks
==> Downloading https://releases.hashicorp.com/tfstacks/0.5.0/tfstacks_0.5.0_darwin_arm64.zip
==> Installing tfstacks from hashicorp/tap
🍺  /opt/homebrew/Cellar/tfstacks/0.5.0: 5 files, 20.1MB, built in 3 seconds
==> Running `brew cleanup tfstacks`...
$ tfstacks -v
0.5.0

You need an alpha release version of Terraform to work with stacks:

$ terraform version
Terraform v1.10.0-alpha20240926
on darwin_arm64

I downloaded an alpha version of Terraform and placed it in a temporary directory. Then I added that directory in the beginning of my path to make sure it will be used first:

$ PATH="/Users/mattias/my/temp/dir/:$PATH"

I did not persist this change outside of the current terminal session.

The Terraform stacks CLI has four different commands:

• tfstacks init: similar to the normal terraform init command, it downloads configuration dependencies and creates a .terraform.lock.hcl file.
• tfstacks providers lock: creates or updates the .terraform.lock.hcl file.
• tfstacks validate: validates the stack configuration, similar to terraform validate.
• tfstacks plan: plans the configuration through HCP Terraform.

Create a New Stack

#

Terraform stack files use the file ending .tfstack.hcl (and .tfdeploy.hcl specifically for deployments). Stacks are written using HCL, but the blocks in these files are not part of the normal Terraform HCL.

In a new directory, create a file named variables.tfstack.hcl and add the following variable declarations:

variable "location" {
  type        = string
  description = "Azure location name"
}

variable "name_suffix" {
  type        = string
  description = "Name suffix for resource names"
}

variable "identity_token" { 
  type        = string 
  ephemeral   = true
  description = "Identity token for provider authentication"
}

variable "client_id" {
  type        = string
  description = "Azure app registration client ID"
}

variable "subscription_id" {
  type        = string
  description = "Azure subscription ID"
}

variable "tenant_id" {
  type        = string
  description = "Azure tenant ID"
}

These variable declarations should look familiar for most Terraform users. However, there is one new feature appearing for the identity_token variable. The variable block takes an optional ephemeral argument. Setting this argument to true makes sure that Terraform does not persist the value to the state file.

Most of the variables will be used to configure the AzureRM provider.

Create a new file named providers.tfstack.hcl with the following content:

required_providers {
  azurerm = {
    source  = "hashicorp/azurerm"
    version = "~> 4.0"
  }

  random = {
    source  = "hashicorp/random"
    version = "~> 3.6.3"
  }
}

provider "azurerm" "this" {
  config {
    features {}
    use_cli         = false
    use_oidc        = true
    oidc_token      = var.identity_token
    client_id       = var.client_id
    subscription_id = var.subscription_id
    tenant_id       = var.tenant_id
  }
}

provider "random" "this" {}

Two things to note:

1. We specify required providers using a required_providers block in the root of the document. This is the same block that we usually specify as a nested block in the terraform block for a normal Terraform configuration.
2. There is a new kind of provider block with two labels (one for the name of the provider, and one for a logical handle to refer to this specific provider). In a regular Terraform configuration the provider block only has one label (for the name of the provider). The provider block has one nested config block where we pass the specific configuration for this provider. The configuration is part hardcoded, and part provided via the variables we defined in variables.tfstack.hcl.

The new type of provider block allows us to configure multiple provider instances of the same provider. We could also use the for_each meta argument inside of the provider block. This would come in handy if we want to create one provider instance for a list of multiple Azure subscriptions or Azure locations.

Create a file named components.tfstack.hcl with the following content:

component "resource_group" {
  source = "./modules/resource-group"

  inputs = {
    location    = var.location
    name_suffix = var.name_suffix
  }

  providers = {
    azurerm = provider.azurerm.this
  }
}

component "storage_account" {
  source = "./modules/storage-account"

  inputs = {
    name_suffix    = var.name_suffix
    resource_group = component.resource_group.resource_group
  }

  providers = {
    azurerm = provider.azurerm.this
    random = provider.random.this
  }
}

Two component blocks are configured. Each component block has the following three arguments:

• A source argument. The value points at a local Terraform module.
• An inputs map that passes values to the variables defined in the Terraform module. We do not specify literal values as input, instead we reference the variables we defined in variables.tfstack.hcl.
• A providers map that passes in provider configurations to the module. We reference the providers we configured in providers.tfstack.hcl using the provider.<name>.<handle> syntax (e.g. provider.azurerm.this).

Next we configure deployments. Create a new file named deployments.tfdeploy.hcl. We will create three deployments (development, staging, production). The deployments look almost identical, so only the development deployment is shown next:

identity_token "azurerm" {
  audience = [ "api://AzureADTokenExchange" ]
}

deployment "development" {
  inputs = {
    location       = "swedencentral"
    name_suffix    = "development"
    
    identity_token  = identity_token.azurerm.jwt
    client_id       = "<your client id>"
    subscription_id = "<your subscription id>"
    tenant_id       = "<your tenant id>"
  }
}

The identity_token block creates an identity token that will be used to authorize the AzureRM provider. This will work since we have already set up the trust relationships between HCP Terraform and Azure (see dynamic credentials with AzureRM above).

The deployment block has one label for the name of the deployment. The name must correspond to one of the names we configured for the federated credentials in Azure. Inside of the deployment block there is an inputs map that takes literal values for the input variables we defined in variables.tfstack.hcl. These values are passed to the components.

The only thing that differs for the staging and production deployments is the name of the deployment block (i.e. staging and production, respectively), and the values passed to the name_suffix variable (again, staging and production, respectively).

Finally, add an orchestration rule to the deployments.tfdeploy.hcl file:

orchestrate "auto_approve" "applyable" {
  check {
    condition = context.plan.applyable
    reason    = "Changes are not applyable"
  }
}

The orchestrate block takes two labels. The first label is the type of orchestration rule, currently only auto_approve can be used. The second label is the name of the orchestration rule.

An orchestration rule allows you to automate the approval of a plan based on the content of a context variable. This variable contains information about the changes that the plan contains. In my orchestrate block I have configured a condition that requires the plan to be applyable. This essentially means that the plan has succeeded without any errors.

Create the Modules

#

We have to create the two modules that our components are referencing. These modules are simple Terraform configurations creating an Azure resource group, and an Azure storage account, respectively. We will not spend time understanding them in any depth since they should be familiar for Azure users.

In the same directory as the stack configuration files, create a new directory named modules with two subdirectories named resource-group and storage-account:

$ mkdir -p modules/resource-group
$ mkdir -p modules/storage-account

Create the resource group module in modules/resource-group/main.tf with the following content:

terraform {
  required_version = "~> 1.6"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

variable "location" {
  type        = string
  description = "Azure location"
}

variable "name_suffix" {
  type        = string
  description = "Name suffix for resources"
}

resource "azurerm_resource_group" "this" {
  name     = "rg-stacks-demo-${var.name_suffix}"
  location = var.location
}

output "resource_group" {
  description = "Azure resource group"
  value = {
    id       = azurerm_resource_group.this.id
    name     = azurerm_resource_group.this.name
    location = azurerm_resource_group.this.location
  }
}

Likewise, create the storage account module in modules/storage-account/main.tf with the following content:

terraform {
  required_version = "~> 1.6"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.6.3"
    }
  }
}

variable "name_suffix" {
  type        = string
  description = "Name suffix for resources"
}

variable "resource_group" {
  type = object({
    name     = string
    location = string
  })
  description = "Azure resource group object"
}

resource "random_string" "this" {
  lower   = true
  upper   = false
  numeric = true
  special = false
  length  = 5
}

resource "azurerm_storage_account" "this" {
  name                = "st${random_string.this.result}${var.name_suffix}"
  resource_group_name = var.resource_group.name
  location            = var.resource_group.location

  access_tier              = "Hot"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

Prepare the Stack For HCP Terraform

#

The current Terraform stack framework requires a file named .terraform-version that contains the version of Terraform that you use to create the stack. This requirement might change when Terraform stacks are released in public beta.

Create this file with the following command:

$ terraform version -json | jq -r .terraform_version > .terraform-version

Next run the tfstacks init command to download providers and create the dependency lock file:

$ tfstacks init

Success! Configuration has been initialized and more commands can now be executed.

You can also validate that everything is working with the validate command:

$ tfstacks validate
Success! Terraform Stacks configuration is valid and ready for use within Terraform
Cloud.

The validate command reports that the stacks configuration is valid and we can use it with Terraform Cloud … Seems like the old name Terraform Cloud lives on here, it should of course be HCP Terraform.

Publish Your Stack as a GitHub Repository

#

You need to publish your stack configuration to a git repository. This is currently the only supported way to work with stacks on HCP Terraform.

I am using GitHub, so I will create a GitHub repository.

First initialize a git repository in the working directory of your Terraform stack:

$ git init
$ git add . && git commit -m "Initial commit"

If you have not configured the GitHub CLI I recommend that you follow the documentation for how to do this to simplify your interaction with GitHub.

Create a new repository on GitHub for your stack using the GitHub CLI:

$ gh repo create \
    --description "Terraform Stacks 101" \
    --private \
    --remote origin \
    --source . \
    --push

All the prerequisites are finally done so that we can move over to HCP Terraform.

Enable Terraform Stacks in HCP Terraform

#

The Terraform Stacks beta is not enabled by default on HCP Terraform. To enable it, go to your organization settings and check the Stacks checkbox:

Creating a Stack In HCP Terraform

#

Create a new project on HCP Terraform. Note that there is a limit of 500 stack resources during the beta period. We will not come anywhere near that in this example.

The name of the project should be the same name you configured for the federated credentials on Azure (in my case it is terraform-stacks-101). You could also provide an optional project description.

Create a new stack in your new project. Notice how workspaces and stacks are separate concepts.

A stack must be connected to a GitHub repository where the stack source code is located. Pick the version control provider that you have configured (see the documentation for details on how to configure a version control provider in HCP Terraform).

Select the repository where your stack source code is located.

Give the stack the same name that you configured when you set up the federated credentials on Azure (in my case it is azure-stack) and provide an optional description of the stack. There are a number of advanced options you can configure, but we will ignore them for now. If you want HCP Terraform to fetch the configuration from the repository when you have created the stack, then select the Fetch configuration after HCP Terraform creates stack checkbox.

HCP Terraform starts the process of preparing the configuration for the stack.

After a while the status changes and you see that the deployments have started rolling out.

Scrolling further down on the page you can see the Deployments rollout section. You can see your three deployments development, staging, and production.

Click on the development deployment to enter the deployment view.

You can further dive into the details by clicking on Plan 1 to see the status of the deployment.

Two things of interest to note:

1. HCP Terraform has gone through a plan operation, followed by an apply operation, followed by a replan operation. What is this replan? This is a new feature concerning partial plans. I will cover that in a different blog post. In this particular case the replan operation makes no difference.
2. We see that our orchestration rule has been applied to automatically approve the change since the plan was successful. If we did not have an orchestration rule, we would have had to manually approve the plan before it would be applied.

Go back to the stack overview page and view the Deployment rollout section. After a few minutes we see that all of the deployments have been rolled out successfully.

You could try to make a change to your stack components (i.e. the underlying modules themselves), push the change to GitHub, and watch as a new deployment rollout kicks in.

Destroy the stack

#

When you are done experimenting with your stack it is time to delete it.

You currently need to delete each deployment separately before you can delete the stack itself. You could force delete the stack, but all stack resources would be left untouched in your Azure environment.

Open one of your deployments. Click on Destruction and deletion in the menu on the left hand side, and select Create destroy plan.

Let the destroy plan run until it reports back that the process was successful.

Repeat this process for each deployment in the stack.

When all deployments have been destroyed it is time to go back to the stack overview page and select Destruction and deletion in the menu on the left hand side, then select Force delete stack azure-stack.

The stack is deleted!

Key takeaways

#

This has been an introduction to Terraform stacks in the context of Microsoft Azure. Key takeaways from this post are:

• Terraform stacks are a new way to scale your Terraform deployments.
• A Terraform stack is a different concept than a HCP Terraform workspace. You can have both stacks and workspaces concurrently.
• A Terraform stack consists of one or more components. Each component is created in a number of instances called deployments.
• HCP Terraform offers a good visibility of all the components and deployments that are part of your stack.
• Orchestration rules allow you to automatically approve a deployment based on conditions that you configure.
• Stacks are still in preview (as of October 2024) and there is a current limit of 500 resources that can be managed through stacks.