Take your testing to the cloud

I have covered testing and validation with Terraform in a few of my latest blog posts, and this post is no different. Well, it is different in the sense that I will discuss testing specifically in Terraform Cloud. To read my previous posts covering the topic of testing and validation with Terraform see the following links:

• A Comprehensive Guide to Testing in Terraform: Keep your tests, validations, checks, and policies in order
• Testing Framework in Terraform 1.6: A deep-dive
• Test permutations with Terraform and GitHub Actions

Introduction⌗

Terraform 1.6 introduced the new testing framework. Apart from being able to run tests locally and in your CI/CD pipelines, the test framework is also integrated with Terraform Cloud.

If you are not familiar, Terraform Cloud is a managed platform provided by HashiCorp for managing Terraform. This includes running Terraform, handling state, gating changes with Sentinel policies, providing projects and workspaces for infrastructure management, providing private registries for modules and providers, and much more¹.

Publishing a module⌗

The first step to run tests in Terraform Cloud is to actually publish a module to your private registry. Remember that we run tests for our modules, not for the regular Terraform configurations building up our applications. This is why there is no way to write tests inside of a given workspace in Terraform Cloud. A workspace is an instance of your regular Terraform configuration, perhaps using one or several of the modules from your registry.

The example module I use in this post to illustrate how testing with Terraform Cloud looks like creates a resource group in an Azure subscription. The directory structure for this module is this:

.
├── README.md
├── main.tf
├── outputs.tf
├── tests
│   └── main.tftest.hcl
└── variables.tf

This module takes the following inputs in variables.tf:

variable "location" {
  type        = string
  default     = "swedencentral"
  description = "Azure location for resource group"

  validation {
    condition = contains([
      "swedencentral",
      "westeurope",
      "northeurope"
    ], var.location)
    error_message = "Use an approved location"
  }
}

variable "name_suffix" {
  type        = string
  description = "Resource group name suffix (i.e. rg-<NAME_SUFFIX>)"

  validation {
    condition     = !startswith(var.name_suffix, "rg-")
    error_message = "Do not add 'rg-' in the name, this is added automatically"
  }

  validation {
    condition     = length("rg-${var.name_suffix}") <= 90
    error_message = "Resource group name too long (should be between 1 and 90 characters)"
  }
}

There are two variables: location and name_suffix. There are validation blocks for each variable to make some basic checks on the validity of the provided value.

The resource group itself is configured in main.tf:

terraform {
  required_version = "~> 1.6.2"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.80"
    }
  }
}

resource "azurerm_resource_group" "this" {
  name     = "rg-${var.name_suffix}"
  location = var.location

  tags = {
    managed_by = "terraform"
    location   = var.location
  }
}

In the terraform block I say that the required Terraform version for this module is at least 1.6.2, and the ~> says that I allow the rightmost value (the patch number) to increase. I have no scientific motivation behind this selection, other than I need at least version 1.6 to run Terraform tests.

The resource group is of type azurerm_resource_group. If you are not familiar with Azure, a resource group is a container for all other resources you would like to create in Azure. Any other resource you create must be placed in a resource group. If you have been working exclusively with AWS this might sound outlandish, but I would argue that this is one of the best features of Azure. At least when it comes to resource organization.

This module has outputs defined in outputs.tf:

output "resource_group" {
  value       = azurerm_resource_group.this
  description = "Resource group object"
}

The output resource_group is the whole resource group object. This is a good practice in Terraform, this allows the module consumer to use any property of the resource. The alternative is to provide separate outputs for any value of interest from the resource, but then you as a module producer needs to forsee every property that might be of interest to your consumers.

We have a module, and we are ready to publish it to our Terraform registry! As of when I am writing this blog post there does not seem to be full support to publish a module and activate testing for the module using the Terraform Cloud REST API. Therefore, I will use the Terraform Cloud web portal to describe the required steps.

First of all we must create a git repository for our module. I will use GitHub, but Terraform Cloud also integrates with GitLab, Bitbucket, and Azure DevOps. The details of how to create a repository is out of scope for this post. One important detail for your git repository is that it should be named terraform-<PROVIDER>-<NAME>. In this case I name my repository terraform-azurerm-resource-group, since the main provider is azurerm. If your module uses many different providers you could instead create your own naming convention as long as the repository name begins with terraform-.

Inside of Terraform Cloud I click on Registry:

I click on Publish and then on Module:

Now I get to specify where my module source code is located. The first time you publish a module you will need to connect to your git repository from Terraform Cloud. I will skip these details in this post. It is relatively straightforward, but if you need guidance you can read the documentation. I select my GitHub connection to move to the next step:

Next I can select the repository where my module source code is located. If you have not followed the naming convention discussed above you will not see your repository in this list. I select my terraform-azurerm-resource-group repository:

In the last step I get to configure how I want my module to be published. There are two ways to administer how new versions of your module are published. The traditional way used git tags, where you pushed a tag to your repository to publish a new version of your module. The new way is to instead publish from a branch. This is what I will use in this case:

I select Branch as the module publishing type, and I specify that I want to publish my module from the main branch. The initial module version is set to 1.0.0. If I scroll down further I can check the box to enable testing for this module:

If you forget to check this box you can always enable testing at a later time. Our module is now published and we end up at the landing page for our module:

Testing a module⌗

In the previous section we published a module to Terraform Cloud. The next step is to configure testing for the module. For this simple module I have written four distinct tests. I will go through them one by one, and in the end show the full test file.

The first test verifies that if I try to add rg- in the beginning of the name_suffix variable I will get an error in the plan operation:

run "do_not_allow_rg_prefix" {
  command = plan

  variables {
    name_suffix = "rg-test"
  }

  expect_failures = [
    var.name_suffix
  ]
}

This is because my module adds this prefix automatically, following my organization naming convention.

The second test verifies that if I try to use a non-allowed location I get an error in the plan operation:

run "do_not_allow_us_location" {
  command = plan

  variables {
    location = "westus"
  }

  expect_failures = [
    var.location
  ]
}

If you have read my post on testing and validation you know that this should be replaced by a policy instead of a test. However, my module does include validations for the location so in that case we should also add tests for it.

The third test verifies that if the name_suffix variable length is too long there is an error in the plan operation:

run "do_not_accept_too_long_name" {
  command = plan

  variables {
    name_suffix = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"
  }

  expect_failures = [
    var.name_suffix
  ]
}

This comes from the fact that the names of Azure resources groups have a limitation that our module should respect.

The fourth and last test runs an apply operation to create an actual instance of the resource group, then it verifies that the correct properties are available in the output named resource_group:

run "verify_output" {
  command = apply

  assert {
    condition     = lookup(output.resource_group, "id", "failed") != "failed"
    error_message = "Output did not have an ID field"
  }

  assert {
    condition     = lookup(output.resource_group, "name", "failed") != "failed"
    error_message = "Output did not have a name field"
  }

  assert {
    condition     = lookup(output.resource_group, "location", "failed") != "failed"
    error_message = "Output did not have a location field"
  }

  assert {
    condition     = lookup(output.resource_group, "tags", "failed") != "failed"
    error_message = "Output did not have a tags field"
  }
}

There are additional test we could write, but this covers the major interface of our module. The full test file named main.tftest.hcl looks like this (the contents of the tests are removed for brevity):

variables {
  name_suffix = "test"
  location = "swedencentral"
}

provider "azurerm" {
  features {}
}

run "do_not_allow_rg_prefix" {
  // ...
}

run "do_not_allow_us_location" {
  // ...
}

run "do_not_accept_too_long_name" {
  // ...
}

run "verify_output" {
  // ...
}

How do we go about running tests for our module in Terraform Cloud? There is one thing we must do. From our module landing page, click on Configure Tests:

Here I can add any environment variables that are needed in order to run my tests. In practice this means I can provide the configuration and credentials required for the providers I am using. In my case I am using the Azure provider, and I will provide a few environment variables to configure the connection to Azure. In the test configuration page I can provide variables, as well as enabling testing to begin with (but we did this earlier when we published the module):

I click on + Add variable and can provide values for the environment variable I want to add:

I provide the Key, the Value and specify that it is Sensitive. Then I click on Add variable. I repeat this for all the required variables. In the end I have all variables configured:

I am ready to trigger my tests. How do I do that? There are two options. You can either push changes to your git repository, specifically to the branch you have selected to publish your module from. In my case it is the main branch. The other option is to trigger tests for your module from your local CLI:

$ terraform test -cloud-run=app.terraform.io/<organization>/resource-group/azurerm

To make this work you need to configure your Terraform CLI to authenticate to your Terraform Cloud organization, the details of this is out of scope for this blog post.

I have done some minor refactoring of my module, so I push these to GitHub. Terraform Cloud picks up the changes and tests are automatically triggered. From the module landing page I can now click on View all tests:

I end up at the test run overview page:

From this page I can click on a test run to see additional details:

There we have it! We have successfully published a module and triggered tests in Terraform Cloud. The next step after this is to publish a new version of our module (assuming we did some noteworthy change, of course), but this is not part of the scope of this blog post.

Generating tests using AI⌗

There is one additional feature related to tests in Terraform Cloud that is currently in beta for Terraform Cloud plus tier customers. We can generate tests for our module using AI. This can be a convenient starting point for old modules that were published before the test framework was available.

To use this feature I go back to the module landing page and click on Generate tests:

A popup warns you that this is a beta feature and references Section 2.5 of the Terraform Cloud User Agreement². Click on Confirm to continue:

Tests are now being generated and we are informed that it could take a while. This is probably true for very large modules, but probably not for small modules like mine:

Once the test generation is done the tests are displayed along with instructions for how to use them and a link to download them as a zip file:

Let’s examine what was generated! Three test files were generated.

In main.tftest.hcl there are three tests. The first test verifies that a few various important properties are set to correct values:

run "resource_group_validation" {
  assert {
    condition     = azurerm_resource_group.this.name == "rg-test"
    error_message = "incorrect resource group name"
  }

  assert {
    condition     = azurerm_resource_group.this.location == "swedencentral"
    error_message = "incorrect location"
  }

  assert {
    condition     = azurerm_resource_group.this.tags.managed_by == "terraform"
    error_message = "incorrect managed_by tag"
  }

  assert {
    condition     = azurerm_resource_group.this.tags.location == "swedencentral"
    error_message = "incorrect location tag"
  }
}

The second test validates that the output resource group is identical to the resource group resource:

run "output_validation" {
  assert {
    condition     = output.resource_group == azurerm_resource_group.this
    error_message = "incorrect resource group output"
  }
}

The third test verifies that the correct values are set for the variables. This might seem like a strange test, but I realize that there might be odd ways of actually modifying them somehow along the way so it makes sense to do this seemingly basic check:

run "variable_validation" {
  assert {
    condition     = var.location == "swedencentral"
    error_message = "incorrect location variable"
  }

  assert {
    condition     = var.name_suffix == "test"
    error_message = "incorrect name_suffix variable"
  }
}

Moving on to variables.tftest.hcl, this file contains a single test:

run "variable_validation" {
  assert {
    condition     = var.location == "swedencentral"
    error_message = "incorrect location variable"
  }

  assert {
    condition     = var.name_suffix == "test"
    error_message = "incorrect name_suffix variable"
  }
}

Wait, this is the same as the last test in main.tftest.hcl! Indeed it is. The AI has repeated itself here. Not sure why this is, but my trained human eye immediately spotted it so I will just ignore this file that the AI has provided for me.

The last file named providers.tftest.hcl was immediately a red flag for me. It contains a single test:

run "provider_validation" {
  assert {
    condition     = terraform.required_version == "~> 1.6.2"
    error_message = "incorrect required_version"
  }

  assert {
    condition     = terraform.required_providers.azurerm.source == "hashicorp/azurerm"
    error_message = "incorrect azurerm source"
  }

  assert {
    condition     = terraform.required_providers.azurerm.version == "3.80"
    error_message = "incorrect azurerm version"
  }
}

What we see here are tests for the terraform block. Does that make sense? I was sure it didn’t make sense, but I decided to give it a try anyway. It turns out it does not make sense. This is nonsense, in fact. So I will gladly ignore this file generated by the AI as well.

If we ignore the obvious duplication in variables.tftest.hcl and the obvious error in providers.tftest.hcl, there was one other issue I noticed. There were no variables blocks generated for these files. This means these tests failed immediately complaining about missing values. So you need to check for this in the generated tests.

Another issue I noticed in all the generated files was that there was no provider block generated. This might, or might not, be required. This all depends on what provider(s) you are using. In my case I use the azurerm provider that requires at least the following configuration:

provider "azurerm" {
  features {}
}

If I don’t include that empty features block then Terraform will complain. So I had to manually add this block to all the generated test files.

Note that you can only generate tests for a module once. You can’t regenerate tests without first removing the module and publishing it again.

Summary⌗

If you are a module producer working in Terraform Cloud you will definitely want to include tests for your modules! In fact, you should do this even if you are not using Terraform Cloud.

In this post I have showed all the steps required to add a module to Terraform Cloud and enabling testing. It does not take a lot of work, but it provides a lot of value.

There is one feature missing that would simplify setting up tests for a module: variable sets. In Terraform Cloud you can add variable sets and connect these to one or several workspaces. A variable set is a collection of one or many variables and environment variables. This is currently not available for module tests, so I will need to configure these variables for each individual module.

The test generation using AI is a beta feature, and it shows. However, as long as you are proficient in the test framework this feature will be a good starting point to get basic tests created. I say that you should be proficient in the test framework, because if you are you will immediately see what you can ignore and what you can use. While writing this post I actually went through publishing this module twice and generating tests twice, and the first generation was a lot better.

If you, the reader of this post, have ideas or questions about the testing framework you want me to tackle in a future post - let me know! Contact me on LinkedIn.