Manage Resource Lifecycle

This post covers the first exam objective of the Terraform Professional certification: Manage Resource Lifecycle.

This exam objective is broken down in the following pieces:

Introduction

#

Managing the lifecycle of your resources is the whole purpose of Terraform. In its simplest form you will write your Terraform configuration, create the resources using terraform apply, and then live happily ever-after. However, in real life things are never that simple.

If you are planning to take the Terraform Professional certification I assume you are already familiar with the basics of terraform init, terraform plan, terraform apply, and terraform destroy. Hence, you can skim through these sections of this post or even skip them completely. The last part of this exam objective covers managing resource state, this part might be of interest to most readers.

Sample Terraform Configuration

#

To illustrate how to manage the resource lifecycle we need a sample Terraform configuration to work with.

Throughout this course I will work primarily with the AWS provider. This is not because I am most familiar with the AWS provider, but because initially that will be the primary cloud provider used in the actual exam. I believe the plan is to offer the exam with focus on different cloud providers in the future, but initially we are stuck with AWS - for better or for worse.

To keep it simple, while still allowing us to cover everything in this exam objective, we’ll use the following configuration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.54.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

resource "aws_security_group" "web" {
  name        = "web"
  description = "Security group for web servers"

  tags = {
    Name = "web"
  }
}

resource "aws_vpc_security_group_ingress_rule" "https" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "Tcp"
}

In this configuration we create two resources, a security group with one associated security group ingress rule.

Store this configuration in main.tf.

Initialize a configuration using terraform init and its options

#

If you have ever used Terraform before (which I assume you have, because you are now reading about the professional certification) you know that the first command you run for all of your Terraform configurations is terraform init.

The terraform init command initializes your configuration by downloading providers and modules, initializes the backend where you will store your state, and creates the .terraform.lock.hcl file (unless it already exists).

Let’s run terraform init for our Terraform configuration:

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "5.54.0"...
- Installing hashicorp/aws v5.54.0...
- Installed hashicorp/aws v5.54.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

The output informs us of what Terraform is doing. Once we have run terraform init we see a new directory named .terraform in our current directory. The contents of this directory is this:

.terraform
└── providers
    └── registry.terraform.io
        └── hashicorp
            └── aws
                └── 5.54.0
                    └── darwin_arm64
                        ├── LICENSE.txt
                        └── terraform-provider-aws_v5.54.0_x5

In the providers directory the required providers are downloaded with the correct architecture for the current system (I am using a mac, i.e. darwin_arm64). In this case we only have the aws provider. If our configuration would have included modules these would have been available under the .terraform directory as well.

If you run terraform init -h you will see all the options that are available for this command. There are a few of these options that might appear on the exam, some of which will be covered in later posts. I want to highlight one option that is important to know: the -upgrade flag.

Currently we have the following terraform block with the required_providers block inside:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.54.0"
    }
  }
}

When we initially ran terraform init, version 5.54.0 of the aws provider was downloaded. Let’s say you want to upgrade the provider version:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.56.0"
    }
  }
}

Just rerunning terraform init will fail:

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/aws: locked provider
│ registry.terraform.io/hashicorp/aws 5.54.0 does not match configured version constraint 5.56.0; must use
│ terraform init -upgrade to allow selection of new versions

This is because you have a .terraform.lock.hcl file where the version is set to be 5.54.0. The lock file currently looks like this:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.54.0"
  constraints = "5.54.0"
  hashes = [
    "h1:kK9b6AtAdfsZ5dQx/C4/bCL7Nx9RzWQlqDKZXRmqIxg=",
    # ... shortened for brevity
    "zh:fbcc3fcd8a4e49763d60c677e1b1b4acc922d8a0f59ae67e38a78345d706ffdd",
  ]
}

To reconcile this error, add the -upgrade flag to the terraform init command:

$ terraform init -upgrade
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "5.56.0"...
- Installing hashicorp/aws v5.56.0...
- Installed hashicorp/aws v5.56.0 (signed by HashiCorp)
Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

Is there a -downgrade flag as well? No. It turns out that downgrading to an older version of a provider is also considered to be an upgrade, so you follow the same procedure.

You should also make sure you know how version constraints for providers work. In the sample Terraform configuration we are working with the provider version has been pinned to a specific version. I consider this to be a best practice, but if you want to loosen the constraint a bit you could allow the patch version of the provider to change. You do this with the following version constraint:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.56.0"
    }
  }
}

If we now rerun the terraform init command with the -upgrade flag we see that the patch version changes:

$ terraform init -upgrade
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.56.0"...
- Installing hashicorp/aws v5.56.1...
- Installed hashicorp/aws v5.56.1 (signed by HashiCorp)
Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

We now have version 5.56.1 of the aws provider.

Generate an execution plan using terraform plan and its options

#

If you are a heavy user of both Terraform and Azure Bicep, you might appreciate the terraform plan command more than others. This is because the corresponding command in Azure Bicep (the what-if operation) leaves you wanting a lot more.

A Terraform plan is a record of changes that would be applied if the current configuration is used in a terraform apply command. There are a number of options for this command that is useful, and might not be something you use every day - so I will go through some of them below.

The basics of terraform plan is to create a plan file, and sometimes convert it to JSON if you need to do something with it (e.g. apply your policy as code to the upcoming changes), and finally apply the plan if it looks OK:

$ terraform plan -out=actions.tfplan
$ terraform show -json actions.tfplan > actions.json
$ terraform apply actions.tfplan

The name of the plan file can be whatever you like. The plan file itself is a binary file, meaning you can’t directly open it to inspect its content. This is why we need to convert it to JSON to be able to see what it contains.

Sometimes it can be useful to plan a single resource, or a few resources that have dependencies between them. You can do this using the -target flag together with the resource address of the resource you want to create a plan for. To do this for our aws_security_group resource we run the following command:

$ terraform plan -target=aws_security_group.web

The output will tell you that using this -target flag is only for exceptional situations, e.g. recovering from errors or mistakes.

Apply configuration changes using terraform apply and its options

#

Once you have a plan file you can tell Terraform to perform the actions that this plan contains with the command:

$ terraform apply actions.tfplan

If you do not have a plan file you can still issue the terraform apply command and Terraform will run a plan operation for you and then ask you to confirm you want to apply the changes. There are situations where you want to just apply the changes without confirming them, this is done by adding the -auto-approve flag:

$ terraform apply -auto-approve

This flag is not recommended for production usage for obvious reasons, but perhaps you will use it in the certification exam just to shave off a few seconds of time!

Something to be aware of for the certification test and real-life is what happens if the plan file you are using is stale? A stale plan file means that something has changed in the infrastructure since the plan file was created. This could be if someone manually edited something outside of Terraform.

To see what this looks like first apply the example Terraform configuration:

$ terraform apply -auto-approve

Next, edit the inbound rule in the Terraform configuration to the following:

resource "aws_vpc_security_group_ingress_rule" "https" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 1
  to_port           = 1024
  ip_protocol       = "Tcp"
}

Note that this change is not something we would actually want to do for our security group, but this is for illustrative purposes only. Now generate a plan file for these changes:

$ terraform plan -out=actions.tfplan

The output will tell us that the ingress rule will be updated in-place, meaning that Terraform expects this ingress rule exists. Before we apply these changes, go to the AWS console and remove the ingress rule from the security group that this Terraform configuration created. Now apply the plan:

$ terraform apply actions.tfplan
aws_vpc_security_group_ingress_rule.https: Modifying... [id=sgr-0775c71f4eff70b37]
╷
│ Error: updating VPC Security Group Rule (sgr-0775c71f4eff70b37)
│
│   with aws_vpc_security_group_ingress_rule.https,
│   on main.tf line 23, in resource "aws_vpc_security_group_ingress_rule" "https":
│   23: resource "aws_vpc_security_group_ingress_rule" "https" {
│
│ InvalidSecurityGroupRuleId.NotFound: The security group rule ID 'sgr-0775c71f4eff70b37' does not exist
│ 	status code: 400, request id: b488889c-d13a-461a-97ed-70081058a3de

The error message clearly indicates what is wrong in this case. However, that might not always be the case. Be aware that if you would be given a plan file you have not generated yourself these types of errors could occur. Try to minimize the time between creating a plan file and applying the changes.

Destroy resources using terraform destroy and its options

#

To remove the infrastructure Terraform has created we use terraform destroy. This command is really just an alias for terraform apply -destroy. We could also generate a destroy plan using terraform plan -destroy -out=actions.tfplan that we can then use with terraform apply. Note that in this case we don’t run terraform apply -destroy or terraform destroy, we just run a regular terraform apply actions.tfplan.

Manage resource state, including importing resources and reconciling resource drift

#

Imagine we want to refactor our infrastructure and move our resources into a module so that we can more easily reuse it elsewhere. This is a common operation and includes manipulating the state file. There are two approaches to manipulating the state file:

• an imperative approach using terraform state CLI commands
• a declarative approach using HCL blocks (import, moved, etc)

A best practice is to use the declarative approach, so that is what we will do here.

We begin from our original Terraform configuration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.54.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

resource "aws_security_group" "web" {
  name        = "web"
  description = "Security group for web servers"

  tags = {
    Name = "web"
  }
}

resource "aws_vpc_security_group_ingress_rule" "https" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "Tcp"
}

The first step is to create the module we want to use. One way to do this is to copy the relevant resources into a new directory modules/security-group. For simplicity we use a single file named main.tf:

// modules/security-group/main.tf
resource "aws_security_group" "web" {
  name        = "web"
  description = "Security group for web servers"

  tags = {
    Name = "web"
  }
}

resource "aws_vpc_security_group_ingress_rule" "https" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "Tcp"
}

To make this a great module we should also parametrize it using variables, but let us skip that step for now because the focus is on migrating state.

In our root main.tf we remove the corresponding resources we just moved to the module and we include a new module block:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.54.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

module "security_group" {
  source = "./modules/security-group"
}

Are we done? Not yet, if we try to apply this configuration we will run into a lot of problems. Terraform will try to delete the resources we had, and at the same time create the module. There will most likely be a conflict between these operations. We need to tell Terraform how our state has changed. We can do this using the moved block, one block for each resource that we moved:

moved {
  from = aws_security_group.web
  to   = module.security_group.aws_security_group.web
}

moved {
  from = aws_vpc_security_group_ingress_rule.https
  to   = module.security_group.aws_vpc_security_group_ingress_rule.https
}

If we now run a terraform plan we see the following:

$ terraform plan
module.security_group.aws_security_group.web: Refreshing state... [id=sg-0c82a39a040399493]
module.security_group.aws_vpc_security_group_ingress_rule.https: Refreshing state... [id=sgr-019bb6e3b7a65318b]

Terraform will perform the following actions:

  # aws_security_group.web has moved to module.security_group.aws_security_group.web
    resource "aws_security_group" "web" {
        id                     = "sg-0c82a39a040399493"
        name                   = "web"
        tags                   = {
            "Name" = "web"
        }
        # (9 unchanged attributes hidden)
    }

  # aws_vpc_security_group_ingress_rule.https has moved to module.security_group.aws_vpc_security_group_ingress_rule.https
    resource "aws_vpc_security_group_ingress_rule" "https" {
        id                     = "sgr-019bb6e3b7a65318b"
        # (8 unchanged attributes hidden)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

The plan summary says 0 to add, 0 to change, and 0 to destroy. We see that Terraform knows the resources have been moved and will thus only update the addresses for these resources in the state file.

Another common operation you might encounter during the certification exam is the need to import an existing resource into the Terraform state. You can do this with the import block.

To test this out first add a new rule to the security group in the AWS console. In this example I have added an ingress rule for port 22 from anywhere. Next I define the same resource in my Terraform configuration (note that I undid the module refactoring steps from before, starting out from the original configuration instead):

resource "aws_vpc_security_group_ingress_rule" "ssh" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 22
  to_port           = 22
  ip_protocol       = "Tcp"
}

Next we add the import block. Note that you must know the security group rule ID of the rule you want to import. This ID has the form sgr-0286259b52ab2e729. You can find the ID in the AWS console where you created the rule. The import block looks like this:

import {
  id = "sgr-0286259b52ab2e729"
  to = aws_vpc_security_group_ingress_rule.ssh
}

We specify the to address in our Terraform configuration, pointing at the resource block we added previously. If we now run a terraform plan we see the following:

$ terraform plan
aws_security_group.web: Refreshing state... [id=sg-0c82a39a040399493]
aws_vpc_security_group_ingress_rule.ssh: Preparing import... [id=sgr-0286259b52ab2e729]
aws_vpc_security_group_ingress_rule.ssh: Refreshing state... [id=sgr-0286259b52ab2e729]
aws_vpc_security_group_ingress_rule.https: Refreshing state... [id=sgr-019bb6e3b7a65318b]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated
with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_vpc_security_group_ingress_rule.ssh will be updated in-place
  # (imported from "sgr-0286259b52ab2e729")
  ~ resource "aws_vpc_security_group_ingress_rule" "ssh" {
        arn                    = "arn:aws:ec2:eu-west-1:<account-id>:security-group-rule/sgr-0286259b52ab2e729"
        cidr_ipv4              = "0.0.0.0/0"
        from_port              = 22
        id                     = "sgr-0286259b52ab2e729"
      ~ ip_protocol            = "tcp" -> "Tcp"
        security_group_id      = "sg-0c82a39a040399493"
        security_group_rule_id = "sgr-0286259b52ab2e729"
        tags_all               = {}
        to_port                = 22
    }

Plan: 1 to import, 0 to add, 1 to change, 0 to destroy.

Note the plan summary saying that 1 resource will be imported.

These were two examples of state manipulations that you might encounter in real-life and during the certification exam. Before the exam, also familiarize yourself with the removed block which is required if you want to remove a resource from your state file.

Summary

#

This post covered the Manage resource lifecycle exam objective. Most of what this exam objective is about should be well known to you when you set out on the professional certification. This includes the terraform init/plan/apply commands. These are basic commands you should have used a lot already.

The last part of this exam objective is more interesting, it covers manipulating your state file and showed you a few example of how to do this in a declarative way.

In the next post we will cover the Develop and troubleshoot dynamic configuration exam objective.