HCP Waypoint Actions: Deep Dive

For a recent HUG event in Göteborg I explored what HCP Waypoint can do, and perhaps the most interesting concept in Waypoint is an action.

In this blog post I will cover HCP Waypoint actions in depth. This post is not an introduction to how you get started with HCP Waypoint. I have written an earlier blog post covering how to set up Waypoint using Terraform.

What are Waypoint Actions?

#

In short: actions allow you to perform day-2 operations on your infrastructure.

There are currently two types of actions available:

• Custom: use this type of action to manually connect to a third-party service using HTTP.
• Agent: execute actions in any environment where you have installed a Waypoint agent.

In the UI there is a third type of action listed: GitHub Actions. For me, this action type is currently disabled (see the following image).

It is likely that there will be more integrations added in the future. However, agent-based actions are very powerful and allow you to integrate with any system you can imagine as long as you put in a little extra effort to set it up.

Examples of what you would use Waypoint actions for:

• Take a backup of a database.
• Restore a backup of a database.
• Update the container image version of an application.
• Run a system scan and post results or alerts to Slack.
• Run an end-to-end test case targeting your production environment.
• Proactively scale out your application prior to an event where you expect an increase in traffic.
• Reboot a virtual machine.
• Clean up the tmp directory in a Linux environment.
• Export logs from a system to blob storage.
• And more!

Working with Waypoint Custom Actions

#

Custom actions use HTTP to interact with third-party services. There is an obvious limitation with this type of action: the third-party service must offer some form of HTTP API you can interact with.

You can manage some parts of custom actions using the Terraform provider for HCP.

Below is an example of configuring a custom action using Terraform. This action sends a message to a Slack webhook:

locals {
  # use your own Slack webhook URL
  slack_url = "https://hooks.slack.com/..."
}

resource "hcp_waypoint_action" "slack" {
  name        = "slack-action"
  description = "Post a message to a slack channel"

  request = {
    custom = {
      method = "POST"
      url    = local.slack_url
      headers = {
        "Content-Type" : "application/json"
      }
      body = jsonencode({
        text = "Hello from $${application.outputs.instance_id}!"
      })
    }
  }

  # if you set up the connection between HCP Terraform and Waypoint
  # in the same Terraform configuration, then you need to make sure
  # that the connection is in place before you create actions 
  depends_on = [
    hcp_waypoint_tfc_config.default,
  ]
}

Unfortunately, at the time of writing you can’t manage action variables using Terraform.

You add actions to the templates for the applications where they should run as follows:

resource "hcp_waypoint_template" "server" {
  name        = "aws-ec2-instance"
  summary     = "An AWS EC2 instance"
  description = "A self-service provisioned AWS EC2 instance. Runs in a dedicated VPC."

  # provide a list of action IDs for this template (and all applications created from it)
  actions = [
    hcp_waypoint_action.slack.id,
  ]

  # other arguments are omitted for brevity
}

In your action you can reference variables from the following locations:

• The application name using application.name.
• The name of the template the application was created from using application.templateName.
• The name of the action using action.name.
• Application outputs using application.outputs.<name> as in the example above (e.g. application.outputs.instance_id)
• Outputs from add-ons installed in the appliction using addon.<instance-name>.outputs.<name>. Note that you need the instance name, not the name of the add-on. I am not sure how you easily obtain this instance name.
• Action variables using var.<name>.

The use of variables are not unique for custom actions, you can also use them for the agent actions that are described in the next section.

Working with Waypoint Agent Actions

#

For agent-based actions you must run one or more instances of the Waypoint agent service. You do this with the HCP CLI.

The agent requires a configuration file (see the following section for details on this file).

You can run the agent as a service on your virtual machine, as a sidecar container in a Kubernetes pod, or as whatever you feel is appropriate for your use case.

To run the agent as a service on an Ubuntu instance, add a service definition file similar to the following:

[Unit]
Description=HCP Waypoint Agent
Wants=network-online.target
After=network-online.target

[Service]
Type=exec
User=ubuntu
Group=ubuntu
ExecStart=/bin/bash /home/ubuntu/start_agent.sh
WorkingDirectory=/home/ubuntu/
Restart=always
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

The ExecStart command invokes the following start_agent.sh script:

#!/usr/bin/env bash

# authenticate to HCP
hcp auth login --cred-file=/home/ubuntu/.config/hcp/credentials/credentials.json

# start the agent
hcp waypoint agent run --config=/home/ubuntu/agent.hcl

To run the HCP CLI you must be authenticated. You can find details of how to set up workload identity federation for EC2 instances running on AWS to allow them to obtain credentials for HCP in my previous blog post on this topic:

Once the agent is started it will poll HCP for any requested action runs. No inbound network access is required for this to work, allowing you to run actions in outbound-only networks.

Agent Configuration File

#

The agent configuration file is written in HCL.

The configuration file contains one or more groups of actions. Each group is configured using the group block:

group "<name>" {
  # actions ...
}

With groups you can set up multiple agents that could handle the same actions. This allow you to scale up your action executions. The name of the group is important.

You configure actions in your groups using action blocks.

There are three types of actions you can configure inside an action block. These are:

• A single run block configures an action that executes a command or runs a script. It can optionally run the command or script inside of a Docker container.

  group "my_group" {
    action "my_action" {
      run {
        command = ["echo", "hello, world!"]
      }
    }
  }
  

• A single http block configures an action that sends an HTTP GET request (currently does not support other HTTP methods).

  group "my_group" {
    action "my_action" {
      http {
        url = "https://my.action.url/"
      }
    }
  }
  

• One or more operation blocks configures an action that could be composed of multiple actions. Each operation block contains a single run or http block or a special status block that reports a status back to HCP Waypoint.

  group "my_group" {
    action "my_action" {
      operation {
        run {
          command = ["echo", "first", "command"]
        }
      }
  
      operation {
        run {
          command = ["echo", "second", "command"]
        }
      }
    }
  }
  

Note how each action has a name (my_action in the examples above). The actions must be configured on HCP Waypoint with the same names, and they need to be connected to the correct agent group.

In my experience it is easier to use a single run block action and invoke a script instead of using any of the other types of actions. This is because in the script you can do anything that you can do with the other action types. However, using the other action types is a valid option and can make the actions easier to understand.

If you need actions to execute on a specific virtual machine or other type of host, then you could add a dedicated group in the configuration file for an agent running on that instance or close to it (e.g. a sidecar container). Add any actions to this group that are specific for this host. You will also need to specifically create this group and add all the actions to it on the HCP Waypoint side (as mentioned before, this is currently not possible using Terraform).

Running actions

#

You can manually trigger agent actions using the HCP CLI:

$ hcp waypoint agent queue -g my-group -i my-action
Operation 'my-action' queued.

Your HCP CLI must be configured for the correct HCP project. Provide the agent group name (in the -g flag) and the action name (in the -i flag).

There currently does not seem to be a way to trigger custom actions using the CLI.

You can also trigger actions from the UI. For actions connected to an application, go to the application landing page and click on the Actions button and select the action to run:

On the pop-up that appears, provide any variables that the action is asking for (none in the following example) and then click on Run:

This particular action extracts some information from an EC2 instance and posts a nicely formatted message to a Slack channel:

This is an agent based action. The agent configuration file is this:

group "${application}" {
  action "echo" {
    run {
      command = ["./slack.sh"]
    }
  }
}

Technically, this is a template file. I replace the application variable by the name of the application. This is because I want to be able to target this specific instance where I am also running the agent. Of course you could run the agent individually and not co-locate it with the application.

The slack.sh script shows that you can interact with any system and perform any type of action:

#!/bin/bash

# Get information from the EC2 instance
TOKEN=$(curl --request PUT \
    --header "X-aws-ec2-metadata-token-ttl-seconds: 1200" \
    "http://169.254.169.254/latest/api/token")

INSTANCE_ID=$(curl --request GET \
    --header "X-aws-ec2-metadata-token: $TOKEN" \
    --silent \
    http://169.254.169.254/latest/meta-data/instance-id)

IDENTITY_DOCUMENT=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/dynamic/instance-identity/document)

AWS_ACCOUNT_ID=$(echo "$IDENTITY_DOCUMENT" | jq -r '.accountId')
REGION=$(echo "$IDENTITY_DOCUMENT" | jq -r '.region')
AVAILABILITY_ZONE=$(echo "$IDENTITY_DOCUMENT" | jq -r '.availabilityZone')

# Get available disk space
DISK_SPACE_LEFT_PERCENT=$(awk 'NR==2 {print 100 - $5}' < <(df -h /))

# Get current date and time
NOW=$(date "+%Y-%m-%d %H:%M:%S")

# Post to Slack channel
read -r -d '' PAYLOAD <<EOF
    {
      "blocks": [
        {
          "type": "header",
          "text": {
            "type": "plain_text",
            "text": "EC2 Disk Space Report (instance $INSTANCE_ID)"
          }
        },
        {
          "type": "context",
          "elements": [
            {
              "type": "mrkdwn",
              "text": "Account: *$AWS_ACCOUNT_ID*"
            },
            {
              "type": "mrkdwn",
              "text": "Region: *$REGION*"
            },
            {
              "type": "mrkdwn",
              "text": "Availability-zone: *$AVAILABILITY_ZONE*"
            }
          ]
        },
        {
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "EC2 instance *$INSTANCE_ID* has $DISK_SPACE_LEFT_PERCENT% disk space left as of $NOW."
          }
        }
      ]
    }
EOF

curl --request POST \
    --header "Content-type: application/json" \
    --data "$PAYLOAD" \
    https://hooks.slack.com/<redacted>

This is just one of a million things your actions could do! Just your imagination sets the limits.

Let me leave you with one last crazy idea: what if you trigger Terraform plans and applies from an action? Could it be done? What would be a good use case here?

Key Takeaways

#

HCP Waypoint actions are powerful. You can use actions to perform any Day-2 operation on your infrastructure.

Custom actions interact with third-parties using HTTP requests. They are limited to interacting with HTTP APIs that are publicly available.

Agent actions execute wherever you need them to execute. This is based on installing an agent close to your infrastructure and have it poll HCP Waypoint for any actions it needs to perform. Agent actions can run scripts and you can build custom integrations with any system that you need to interact with.

Currently there is very limited support for managing actions using Terraform. Hopefully this will change in the near future.