I assume you are reading this blog post because you are a GitHub and Terraform user, and you want to provision infrastructure to a target cloud provider (I use Microsoft Azure as an example in this post).
Perhaps you are reading this blog post because you want to reduce the cost of running Terraform.
There are many options for how to run Terraform. Personally I think there are too many options1. Some come with a cost. You are already paying for your target platform (e.g. Microsoft Azure) and your version-control system (e.g. GitHub), do you have to pay for Terraform as well?
The BSL license aside2 Terraform is available for free. You can build your own Terraform automation on GitHub, and utilize services on your target platform for additional features offered by Terraform automation platforms (e.g. HCP Terraform). At the very least you need to store the Terraform state file somewhere.
You are reading the first part of a short series exploring how you can build Terraform automation on GitHub for a fraction of the cost of other platforms3.
The example discussed below is available on my GitHub.
The GitHub provider for Terraform#
The GitHub provider for Terraform can manage individual repositories, organizations, and even enterprises.
Add the GitHub provider to the list of required providers4:
terraform {
required_providers {
github = {
source = "integrations/github"
version = "~> 6.0"
}
}
}The GitHub provider is a partner-provider, published under the integrations namespace. I’m pointing this out as some Terraform users go through their entire life without being exposed to providers outside of the official hashicorp namespace.
Authenticate the GitHub provider with one of three options:
- Install the GitHub CLI and run
gh auth login. I will use this method. - Create a personal-access token (PAT) on GitHub. Set the token as an environment variable named
GITHUB_TOKEN. - Use a GitHub App installation for your GitHub user or organization. Configure the provider with the app ID, the installation ID, and an app PEM file.
For your own labs, consider using the GitHub CLI for simplicity. For production automation scenarios, consider using a dedicated GitHub app.
Introducing Terraform landing zones on GitHub#
I have written and spoken about the concept of landing zones before.
I now want to build a Terraform landing zone on GitHub. This includes everything required to execute Terraform to provision cloud infrastructure to a target platform.
At a minimum, a Terraform landing zone on GitHub includes:
- A Terraform backend for state storage.
- A GitHub repository.
- A starter Terraform configuration. At a minimum this should include provider and backend configuration.
- Trust relationships with a target platform (e.g. Microsoft Azure) with permissions to provision resources to this platform.
- GitHub Actions for common Terraform workflows.
These components are visualized below:
This is not rocket science, but it is a very powerful setup!
This setup is a great start. In this post I will not dive deeper than this. In a future post I will consider how to scale this approach and how to provide built-in governance.
Building a Terraform landing zone on GitHub#
I now know what I want my Terraform landing zone on GitHub to consist of. Below I will discuss each component closer.
A Terraform state backend#
A state backend on Azure consists of a storage account along with a storage blob container. In addition there will be RBAC resources to manage authorization, and possibly network resources to manage network access to the backend.
You can provision a dedicated state backend for each landing zone. This provides the best possible separation of concern and greatly decreases the blast radius. For simplicity I use a shared state backend for all landing zones.
The state backend has a long lifecycle and is often provisioned and managed by a platform team.
GitHub repository#
The GitHub repository is the heart of the landing zone. This is where the Terraform source code is stored, and where the GitHub Actions workflows that will run Terraform are configured.
Personally I like a setup where everyone in my organization has read-access to every repository. In addition everyone should be able to propose changes to the code through a pull-request. This is inner-source thinking. Only the team who owns the landing zone should have administrator rights to the repository.
The GitHub provider for Terraform allows you to configure repositories in a number of ways. The exact configuration of the repository resource itself is not crucial for a successful landing zone, the important details are around what the repository contains.
To manage rules for multiple repositories in a simple manner we can introduce repository custom properties. This is a topic for a later blog post.
Sample Terraform configuration#
The GitHub repository in the landing zone comes with a sample Terraform configuration. At a minimum this Terraform configuration should include details about required providers and the state backend configuration.
The Terraform state backend configuration can be set using environment variables, but if possible it is a good idea to be explicit by using a backend.tf file with a configured backend block:
# backend.tf
terraform {
backend "azurerm" {
use_oidc = true
use_azuread_auth = true
storage_account_name = "terraformstate"
container_name = "state"
key = "state/happy-unicorn/terraform.tfstate"
}
}An alternative to creating the sample Terraform configuration using Terraform is to use a template repository on GitHub. The template repository can contain boilerplate Terraform code that each landing zone starts off with.
Trust relationship#
The trust relationship between GitHub and your target platform builds on OIDC and workload identities. I have covered this in different variations in multiple blog posts before. Workload identities are so powerful that I will keep covering it over and over again!
In my example I want to provision infrastructure on Microsoft Azure. There are two possibilities:
- I can use a managed identity (an Azure resource).
- I can use an app registration and service principal (Entra ID resources).
Using an app registration and service principal is often preferred, at least in my experience. One thing I like about them is that they support flexible federated identity credentials.
Each landing zone comes with two app registrations and two service principals:
- One is used for running
terraform planwhen opening a pull-request targeting the main branch. This identity only requires read access on my Azure subscription. - One is used for applying changes through
terraform applywhen changes are merged to the main branch. This identity requires write access on my Azure subscription.
Apart from read or write access to the subscription, both identities require permissions to manage blobs in the state backend.
Do not use a single identity with read and write permissions for everything. Separating the concern in this way allows the workflow to run even when someone outside the team triggers the workflow (i.e. an inner-source contributor) without giving them write privileges to your cloud environment.
For Azure, this trust relationship requires a few environment variables for the connection to work (i.e. client ID, subscription ID, and tenant ID). These are set as GitHub Actions secrets.
GitHub Actions workflows#
The repository should come with a few GitHub Actions workflow for common Terraform operations. These workflows come as files in the .github/workflows directory, in addition to the sample Terraform configuration covered earlier.
At the very least there are two important workflows to include:
- Run
terraform planwhen opening a PR targeting the main branch. This workflow will also support manual triggers on a feature branch. - Run
terraform applyto apply changes merged into the main branch.
When provisioning Terraform files to a GitHub repository using Terraform you do not want to track changes to the contents of these files. If you do, you might inadvertently overwrite changes made by others to these files.
When a team receives their landing zone they might change how they structure the Terraform configuration in their repository. The sample Terraform configuration should be considered “fire and forget” resources.
Package the landing zone as a Terraform module#
The landing zone should be packaged as a Terraform module. This allows easy reuse for creating multiple landing zones.
In the sample repository the landing zone is packaged as a local module and it is used as in the following code snippet:
module "happy_unicorn" {
source = "./modules/azure-landing-zone"
github_owner = var.github_owner
name = "happy-unicorn"
state_backend = <<-HCL
terraform {
backend "azurerm" {
use_oidc = true
use_azuread_auth = true
storage_account_name = "${azurerm_storage_account.state.name}"
container_name = "${azurerm_storage_container.state.name}"
key = "state/happy-unicorn/terraform.tfstate"
}
}
HCL
}This module accepts the state backend configuration as an input variable along with the name of the landing zone and the GitHub owner (username or organization name where the repository should be created).
Key Takeaways#
The relatively simple example outlined above is the foundation for a secure, scalable, and cheap Terraform automation platform.
You can build a similar system using other version-control systems (e.g. Gitlab). The key takeaway is that it is easy to build your own platform for automating Terraform.
If it is this easy, why would you pay for a platform such as HCP Terraform? One of the primary reasons might be enterprise support. If you are running mission-critical workloads you want someone (preferably someone that is not you) to provide equally mission-critical support if needed.
If you build your own automation platform you are responsible for running it. If you have the engineering manpower to do it, then go ahead!
Organizations suffer from the Paradox of Choice. ↩︎
If you are building a product that offers Terraform in a SaaS-consumption manner, you will have to pay to use Terraform. That is not what we are doing in this blog post. ↩︎
This post won’t go into pricing details, so how can I say that this will be cheaper? Simply because this costs close to zero on top of what you are already paying for GitHub and your target platform. ↩︎
At the time of writing the latest available version is 6.9.1. Make sure to use a version constraint that makes sense to your needs and your environment. ↩︎
