Terraform Ephemeral Resources

Terraform 1.10 introduced a new concept called ephemeral resources.

An ephemeral resource is not persisted to the state file. Let that sink in for a moment!

Ephemeral resources solve the long standing problem of secret values being persisted to the state file in plaintext. Your secret values are no longer in danger if your state file is lost. Well, this is true if we fast-forward a few weeks, months, or maybe a year.

Initially, only four selected providers have two or more supported ephemeral resources to start with. These providers and corresponding resources are:

• Amazon Web Services (i.e. aws)
  • aws_secretsmanager_secret_version
  • aws_lambda_invocation
  • aws_kms_secrets
• Microsoft Azure (i.e. azurerm)
  • azurerm_key_vault_secret
  • azurerm_key_vault_certificate
• Google Cloud Platform (i.e. google)
  • google_service_account_access_token
  • google_service_account_id_token
  • google_service_account_jwt
  • google_service_account_key
• Kubernetes (i.e. kubernetes)
  • kubernetes_token_request
  • kubernetes_certificate_signing_request

This list will be extended over time.

If you have an idea of what ephemeral resource type should be prioritized next, create an issue on the corresponding provider GitHub repo:

• AWS
• Azure
• GCP
• Kubernetes

Declaring an ephemeral resource

#

Ephemeral resources introduce a new root level block in HCL: the ephemeral block.

The general format of this block looks a lot like that of a normal resource block:

ephemeral "<resource type>" "<resource name>" {
  # attributes, meta-arguments, nested blocks
}

As with normal resources, the supported attributes and nested blocks varies depending on the ephemeral resource type.

Using an ephemeral resource

#

Referencing an ephemeral resource is similar to how you reference a data source. References start with the ephemeral. prefix.

Given the following ephemeral resource:

ephemeral "azurerm_key_vault_secret" "db" {
  # attributes
}

You can reference attributes from this ephemeral resource by ephemeral.azurerm_key_vault_secret.db.<attribute>.

There are a few rules associated with where you can use and reference attributes of ephemeral resources. The following locations/objects are supported:

• In another ephemeral resource block
• In local values
• In ephemeral variable blocks (see below)
• In ephemeral output blocks (see below)
• Configuring providers in the provider block (see the following example)
• In provisioner and connection blocks of a normal resource

The reason for these rules is that none of the objects above are persisted in the state file. If you would be able to reference the ephemeral resource somewhere where it would be stored in the state file, then the whole purpose of the ephemeral resource would be defeated.

Using an ephemeral value in a local value makes the local value into an ephemeral local value. You can’t create an explicit ephemeral local value, you can only implicitly create them by referencing ephemeral values in local values.

The following code snippet¹ is an example of how ephemeral resources can be used:

# read the secrets from aws secrets manager
ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "<secret id>"
}

locals {
  # decode the json secret value
  credentials = jsondecode(ephemeral.aws_secretsmanager_secret_version.db.secret_string)
}

# configure the postgresql provider using the credentials
provider "postgresql" {
  host     = "<postgres endpoint>"
  port     = 5432
  username = local.credentials["username"]
  password = local.credentials["password"]
}

In the example above the ephemeral resource of type aws_secretsmanager_secret_version is used to create a new secret version for a PostgreSQL database password. The password is JSON decoded in a local value. Finally, values from the JSON decoded secret is used to configure the postgresql provider.

An ephemeral resource supports the common meta-arguments that are supported for normal resources:

• depends_on to make hidden dependencies explicit
• count to create a number of identical ephemeral resources
• for_each to create one ephemeral resource for each value of a set or map
• provider to use a provider alias
• lifecycle to hook into the lifecycle of the ephemeral resource

However, ephemeral resources do not support the provisioner meta-argument. You should avoid using this meta-argument anyway, so no problem!

The lifecycle of an ephemeral resource

#

Ephemeral resources behaves different than what you are used to for normal resources and data sources.

Ephemeral resources are opened (or read) at the point when Terraform requires the values of the ephemeral resource. When the values are no longer needed, Terraform closes the ephemeral resource. What opening/closing an ephemeral resource means differ depending on the type of ephemeral resource.

A concrete example is a secret in HashiCorp Vault. Opening/reading a secret in Vault means obtaining a lease for this secret. Closing the secret in Vault means explicitly ending the lease.

The most important aspect of the lifecycle of an ephemeral resource is that at no point in time is the value persisted to the state file.

Ephemeral variables and ephemeral outputs

#

Apart from ephemeral resources, there are two related concepts:

• Ephemeral variables
• Ephemeral outputs

Ephemeral variables

#

If you have read my blog in the past few weeks you have already seen the use of ephemeral variables in the context of Terraform Stacks.

Ephemeral variables are variables that are not persisted to the state file. You can declare a variable as sensitive, that tells Terraform to avoid printing the value in logs. However, ephemeral variables are guaranteed to not appear in the state file (sensitive variables can appear in the state file).

You declare a variable as ephemeral using the ephemeral argument:

variable "my_ephemeral_variable" {
  type      = string
  ephemeral = true
}

One small counterintuitive thing is that ephemeral variables do support the default argument:

variable "my_ephemeral_variable" {
  type      = string
  ephemeral = true
  default   = "an ephemeral value"
}

You can reference an ephemeral variable in one of the following objects:

• In another ephemeral variable block
• In a local value
• In an ephemeral resource block
• In an ephemeral output block (but not in the root module)

Ephemeral outputs

#

Ephemeral outputs are only allowed in a module other than the root module. You can pass the value of an ephemeral output to an ephemeral input of a different module.

An ephemeral output is declared by adding the ephemeral argument to the output block:

output "my_ephemeral_output" {
  ephemeral = true
  value     = aws_secretsmanager_secret_version.this.secret_string
}

Ephemeral outputs can be used in one of the following places:

• As a value passed to another module’s ephemeral variable blocks
• In an ephemeral resource block
• Passed as an ephemeral output from the parent module (unless it is the root module, no ephemeral outputs are allowed in the root module!)

Summary

#

Ephemeral resources, variables, and outputs seem like a great addition to Terraform.

I think they will be used a lot going forward.

The main point to take away from this post is that ephemeral resources/variables/outputs remove the corresponding values from appearing as plaintext in the state file. This has been a long running problem since the birth of Terraform. The Terraform state file still contains sensitive information in the form of a complete map of your cloud infrastructure, so you will still need to protect your state file like before.

Read the official documentation to learn more about ephemeral resources.