Using HashiCorp Vault to generate temporary credentials to Azure from GitHub Actions

Introduction

#

The HashiCorp Cloud Platform (HCP) simplifies working with many of the great HashiCorp products that are available. One of these products is Vault. Vault is a platform for connecting your application secrets (or any kind of static or dynamic secrets for that matter) with the users and applications that require access to them. Secrets can be stored almost anywhere, and you can authenticate to Vault with a multitude of options.

The possibilities that Vault opens up are almost endless, but you have to start somewhere! In this post I will:

• Create a new Vault cluster on HCP using Terraform (another great tool from HashiCorp).
• Configure my Vault cluster with the Azure secrets engine and the AppRole authentication method.
• Create a GitHub Actions Workflow that connects to my Vault cluster, asks for temporary credentials to an Azure Service Principal, uses these credentials to authenticate the Azure CLI.

This post is not a gentle introduction to any of the technologies that I will use, but hopefully it will whet your appetite to learn more about everything!

Prerequisites

#

If you want to follow along what I do in this post there are a number of prerequisites that must be in place. I assume you already have git installed, but apart from that you will also need to:

• Sign up for a HashiCorp Cloud Platform account here
• Sign up for an Azure account here
• Install the Azure CLI following these instructions
• Install the GitHub CLI following these instructions
• Install Vault following these instructions
• Install Terraform following these instructions

Create an Azure service principal for Vault

#

For Vault to be able to create and delete Azure service principals, as well as assigning roles to these service principals, it will require a service principal of its own with an appropriate Azure role assignment as well as Microsoft Graph API permissions.

There are three steps I need to perform:

1. Create the service principal.
2. Assign the service principal an Azure owner role on my Azure subscription.
3. Assign the service principal Microsoft Graph API permissions to allow it to create service principals on its own.

Create a service principal and assigning Azure roles

#

I will manually create a service principal for Vault using the Azure CLI and give it the owner role on my subscription with a single command (i.e. step 1 and 2 from above):

The command creates a service principal named hcp-vault with the owner role on my subscription. The output includes details that I will provide to Vault when I configure the Azure secrets engine in a later step. For now I store the output somewhere safe for later.

Assign Microsoft Graph API permissions

#

The service principal needs permission to create new service principals on its own. This means I have to give it an application permission (not a delegated permission) named Application.ReadWrite.All. I can do this with az ad app permission add:

The strange looking GUID I provide for the --api flag corresponds to Microsoft Graph API, and the other GUID I provide for the --api-permissions flag corresponds to the Application.ReadWrite.All permission. Since this is an application permission (i.e. the application will act on its own without a signed-in user) this requires consent from an Azure AD administrator. Lucky for me I am my own Azure AD administrator, so I can consent using az ad app permission admin-consent:

Now I have an Azure service principal that Vault can use to do what I want it to do.

Setting up a Vault cluster on HCP

#

I will use Terraform¹ to set up my Vault cluster on HCP. For this purpose I will use the HCP Terraform provider. The full Terraform configuration to set up a Vault development cluster on HCP looks like this:

Let me explain some of the contents of this configuration:

• On line 10 I include an empty provider configuration block for the HCP provider. This is where I could supply the required credentials needed to authenticate to HCP, but I will set the required values as environment variables instead.
• On line 12 I create a hcp_hvn resource. This is a HashiCorp Virtual Network resource. Placing our cluster resources in a virtual network opens up the possibility to peer your own network together with it, to allow secure communication between your applications and the Vault cluster. I will not take that step in this post, however.
• Finally, on line 19 I create the cluster itself. As can be seen on line 23 I set public_endpoint to true, this is not recommended for production scenarios but I use it here to keep the level of this post reasonable.

To be able to work with HCP from Terraform I must set up a HCP service principal to authenticate with. The steps to create a HCP service principle are as follows:

1. I sign in to HCP and go to my organization.
2. I click on Access control (IAM) in the left-hand menu.
3. I click on Service principals in the new left-hand menu that appeared.
4. I click on the + Create service principal button.
5. I provide a name for my service principal and assign it the Contributor role.
6. I click on the Save button and I am presented with a client ID and a client secret.

With my HCP service principal in place I set the client ID and client secret as environment variables (HCP_CLIENT_ID and HCP_CLIENT_SECRET, respectively) in my terminal where I will run Terraform:

I now have my Terraform configuration and I have the required credentials to work with HCP from Terraform. If you are familiar with the Terraform core workflow you know that the first step is the initialization step with terraform init:

The initialization step installed the HCP provider in my working directory. I run terraform plan to generate a plan for what resources Terraform will create:

Terraform reports that two resources will be created, and that is the correct amount of resources that I expect. Since the plan looks good to me I go on and apply the plan using terraform apply:

It takes a while to create the cluster (16 minutes and 6 seconds in my case, plus 2 minutes and 26 seconds for the network), so you can safely take a coffee break while you wait ☕️.

Configuring the Vault cluster

#

Now it is time to configure my Vault cluster.

First of all, I need to set some environment variables with details that will allow me to communicate with my Vault cluster. To find what these environment variables are and what values they should have I follow these steps:

1. I sign in to HCP and click my way to my Vault cluster.
2. I click on Generate token under the New admin token header to generate an admin token for my cluster.
3. I click on Access Vault and then choose Command-line (CLI).
4. I copy all of the commands that are shown and paste them into my terminal:

Now I can verify that I have a working connection to my cluster using vault status:

We need to enable the Azure secrets engine for Vault to be able to work with Azure. I run the vault secrets enable command to enable the secrets engine:

Then I run a vault write command to write configuration for the Azure secrets engine and provide information for my Azure service principal that I created in the previous section:

Next I need to specify what should happen when GitHub requests a temporary Azure service principal. To keep things simple I will let the temporary service principal become a contributor on my Azure subscription. To configure this I will use vault write to create a role named github-role in the Azure secrets engine:

I provided the list of Azure roles (with a single entry) inline, but I could also have stored it in a file. Note that I specified ttl=5m in the command, this tells Vault that the Azure service principal that is created should be deleted after 5 minutes, i.e. this is what makes the credentials (and the service principal itself) temporary.

I want GitHub to be able to communicate with my Vault cluster so GitHub will require a Vault token to do so. There are a few options on how to proceed. The method I will use is to activate the AppRole authentication method, and create an AppRole for GitHub with an appropriate policy attached. The policy I will use is this:

This policy says that GitHub will only be able to run read operations on the azure/creds/github-role path in my Vault cluster. I create the policy in Vault using vault policy write:

I enable the AppRole authentication method using the vault auth enable command:

I create an AppRole named github for GitHub with the Vault policy I created before:

The final step is to extract a role ID and a secret ID for the github AppRole, these values will be provided as secrets to my GitHub repository. First I need the role ID:

Then I need the secret ID:

Now I am done configuring Vault! I copy the role ID and the secret ID to a safe place, I will set them as repository secrets in GitHub later. The next step is to use Vault from GitHub!

Setting up a GitHub repository

#

All the prerequisites are taken care of, and now it is time for an imaginary developer team to set up a new GitHub repository and build a GitHub Actions workflow that will use Vault to obtain a temporary service principal.

For this imaginary scenario I will only use the temporary service principal to list all resources groups in my subscription and count how many they are.

I start in an almost empty local git repository². I create a new GitHub repository using the gh command line tool:

To provide the required secrets to GitHub I run the command gh secret set once each for the following keys:

• VAULT_ROLE_ID with the role ID I obtained from Vault.
• VAULT_SECRET_ID with the secret ID I obtained from Vault.
• TENANT_ID with the value of my Azure AD tenant ID.
• SUBSCRIPTION_ID with the value of my Azure subscription ID.

I add a new GitHub Actions workflow to .github/workflows/list-resource-groups.yaml with the following content:

To communicate with Vault I have used the Vault action. This action provides many different ways of authenticating to Vault, but since I set up an AppRole I use the approle method. I provide the role ID and secret ID to vault for authentication, and I specify what secrets I want to extract from Vault in the secrets parameter. The syntax azure/creds/github-role client_id | AZURE_CLIENT_ID means “read the secret named azure/creds/github-role from Vault, take the client_id value from the output and make it available as an environment variable named AZURE_CLIENT_ID”.

The rest of the workflow authenticates the Azure CLI using the temporary service principal credentials together with the subscription ID and tenant ID I stored as secrets before, and finally it lists and counts the number of resource groups that I have in my subscription.

Since I included a manual workflow trigger I can start the workflow using the gh command line tool:

Once the workflow has finished I can verify that the workflow worked from the result (looks like I have 6 resource groups):

If I wait for five minutes then the service principal that Vault created will be removed. No valid credentials to Azure remains! Note that it takes a while (around 20-30 seconds) to create the service principal and associated secret, so this is not a method you would use for anything that require immediate credentials.

Summary and outlook

#

This post contained a lot of technicalities! What have we actually achieved here? We have an embryo for a GitHub Actions workflow that obtains temporary credentials to perform actions in Azure. These credentials are short-lived, and there is no need to have long-lived service principals with secrets that have to be rotated.

One could argue that we have now moved the problem to the role ID and secret ID that we used to authenticate to Vault instead of the Azure service principal client secret that we would otherwise have to protect. That is true in some way, and if this was the only thing we used our Vault cluster for then this would indeed be a valid criticism (and also an expensive solution because HCP Vault is not free of charge). However, the idea is to use the Vault cluster for all your secrets management. What I demonstrated here was for Azure, but you might also require temporary credentials to a database, or to AWS, or for SSH-access to your build-agents. As with many great tools it is often overkill to use it for a small piece of your overall architecture, instead you should utilize it everywhere.

How can you make further improvements on this architecture? In a production scenario I would have set up my own VNET in Azure, and I would have peered this network with my HCP network. With that done I can turn off public access for my Vault cluster, and keep all traffic to my cluster inside my own network. This would also require me to use my own build agents in GitHub. I would also look in to different methods of authenticating GitHub to Vault, instead of using the AppRole method. If no other method seems to be suitable I would at least configure a TTL on the secret ID and have GitHub acquire a new one periodically.

There is much to learn, and my journey with HashiCorp Vault (and HCP Vault specifically) have only just begun. I am intrigued by the possibilities!