3 ways secrets management improves monitoring & observability

Secrets management for Sensu Go, and the HashiCorp Vault Integration

Monitoring — by its very nature — requires privileged access to internal and external services. In order to safely maintain visibility into critical systems, it’s vital to have some form of secrets management to manage authentication credentials (AKA, “secrets”), including passwords, keys, APIs, tokens, and any other sensitive pieces of information in your IT infrastructure.

With more workloads being offloaded to the cloud, we’ve moved away from traditional infrastructure models with trusted network perimeters. While we can no longer implicitly rely on those secure network perimeters, there are several excellent services that offer a trusted source of identity and secrets management. In this new (often multi-cloud) reality, a dynamic approach to secrets management is also needed to automate previously manual processes and ensure organizational security.

Secrets management can improve monitoring and observability in three critical ways, each of which we’ll dive deeper into below:

1. Removes barriers to self-service access to monitoring
2. Helps you meet security and compliance requirements
3. Improves automation

In this post, we’ll take a look at secrets management (what it is and the value it brings your organization), highlighting HashiCorp Vault as our main example. We’ll also demo how secrets management in Sensu works with our HashiCorp Vault integration, as well as our environment variable secrets integration, which allows you to integrate with the secrets management provider of your choosing, including Google Cloud Secrets Manager, AWS Secures Manager, Azure Key Vault, Kubernetes Secrets, CyberArk, and more.

Secrets management: the what and why

Secrets management checks a number of boxes when it comes to security and compliance. The real value of secrets management comes from centralized management of those secrets, which allows for accountability — enabling you to audit who or what systems had access to various secrets at different points in time — and allows you to introduce automation into previously manual processes. That is, you can automate how you manage access to secrets, whether they’re passwords, API tokens, or TLS certificates. Once you have a management control plane in place, you can automate the rotation of those secrets, setting them to expire or be replaced after a set amount of time, paving the way for “dynamic secrets” workflows in which your systems are rotating secrets more frequently than users would, improving the security posture of your organization.

It’s amazing to see how many secrets management solutions are available today – from dedicated solutions like HashiCorp Vault and CyberArk, to built-in offerings in essentially every modern cloud computing platform. With so many options to choose from, it’s important to understand which tool(s) are right for your organization’s current and future needs. I found a comment in a recent blog post by Drew Rothenstein at Coinbase to be quite helpful on this topic:

While some platforms may state that they have other features such as […] secret/config. management […]: the reality is that these generally do not work for larger scale installations without intense investments either in forking / customization or through integrations and separation.

For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management. These primitives are generally not meant, designed, or built for hundreds of engineers on tens of teams and generally do not include the necessary controls to be able to sanely manage, own, and operate their applications. It is extremely common for folks to separate their secret and config management to a system that has stronger guarantees and controls (not to mention scaling).

Beyond the primary benefits of dedicated secrets management solutions, there are also far-reaching impacts on Digital Transformation (DX) initiatives. More and more organizations are adopting and embracing practices such as infrastructure as code, and “shifting left” by offering developers self-service access to critical infrastructure like compute and monitoring. Abstracting secrets away from underlying code or declarative configuration templates drastically reduces friction around onboarding new teams and extending the reach of these initiatives.

From a monitoring perspective, sanitizing monitoring configuration in a way that makes it shareable across your organization is critical. By removing secrets from your monitoring configuration, you can share your solutions more easily within your organization, streamlining the onboarding process for new users in a safe, secure, and compliant way.

Sensu’s secrets management simplifies the process of sanitizing your monitoring configs, making it easier (and safer!) to share your solutions. Sensu secrets resources reference externally provide secret values — so those secrets never appear directly within your monitoring configuration. These resources are portable across Sensu namespace and clusters, so you can easily share your configs with other team members (and even other members of the Sensu Community!).

Here’s an example of PagerDuty secrets within a Sensu configuration:

In addition to HashiCorp Vault — which we’ll cover in the next section — Sensu supports any secrets management provider (Google Cloud Secrets Manager, AWS Secures Manager, Azure Key Vault, Kubernetes Secrets, and CyberArk, to name a few) via our environment variable secret integration.

As an example, let’s say you have a Linux-based environment like RHEL or CentOS with Sensu running as a Systemd service. You can set environment variables in your development or staging environment (e.g. by modifying /etc/default/sensu-backend). Those environment variables will be exposed to your Sensu backend in its execution environment, for which Sensu can then use handler definitions or even distribute secrets to the Sensu agents for checks.

Here’s how Kubernetes Secrets would work with the Sensu Go env provider:

To learn more about using Sensu Secrets Management with Kubernetes Secrets, please visit this recent Discourse post.

It’s worth noting that Sensu won’t allow you to distribute secrets in checks unless you’ve configured TLS-encrypted transport and enabled certificate-based agent authentication (mTLS) between the agent and backend. Sensu will never transmit secrets over insecure channels — secrets must be encrypted in transit with a strong authentication mechanism in order to be transmitted to the Sensu agent. In that way, you can safely and securely share secrets with Sensu agents to access monitoring data while satisfying organizational security and compliance requirements.

In the next section, we’ll take a look at secrets management with HashiCorp Vault, plus a demo of how it works with Sensu.

Secrets management with HashiCorp Vault

As touched on earlier, a secrets management tool should enable you to not only protect your sensitive information but also automate your workflows — removing users from the manual process of updating secrets, for example. HashiCorp Vault enables you to centrally store, access, and deploy secrets across applications, systems, and infrastructure, as well as audit access, automatically. Recognizing the shift from static infrastructure (with clear network perimeters) to cloud (with no clear network perimeters), HashiCorp developed Vault as an identity-focused solution to secrets management. Acknowledging the cloud is inherently “low trust,” HashiCorp aims to keep secrets secure by taking an identity-focused approach to secrets management — “Securing the infrastructure and application services themselves through a trusted source of identity and secrets management.”

The HashiCorp Vault operating model of identity-based secrets management. Source.

Vault introduces the concept of dynamic secrets (part of HashiCorp’s “Cloud Operating Model”); instead of a hard-coded secret in a config file, secrets are provisioned via a lease with a set Time to Live (TTL) value. You can issue secrets on a time interval of your choosing and rest assured that your applications will keep up. HashiCorp positions Vault as the standard for non-interactive use cases — that is, they’ve made it so you can automate secrets management as much as possible, leaving you to focus on tasks that can’t be automated.

Benefits of creating dynamic secrets per cloud service include:

• You can make secrets short-lived as necessary
• In the event of a breach, you can quickly revoke secrets
• You can audit all actions on a per-secret basis

For more on identity-based security and dynamic secrets, check out HashiCorp’s whitepaper.

Secrets management with Sensu and HashiCorp Vault (plus a demo)

Sensu provides direct integration with HashiCorp Vault, with support for Vault’s token authentication method and certificate authentication method.

Sensu offers direct and secure access to HashiCorp Vault secrets. In keeping with the principle of least privilege, we keep the footprint that requires that access small; while you may have hundreds or thousands of nodes in your Sensu deployment, only the sensu-backends need access to HashiCorp Vault. We support both root token authentication (while not preferred for a production environment, it’s helpful in dev and test environments) as well as certificate-based authentication to HashiCorp Vault. You can scope which secrets Sensu has access to within Vault and rotate the secrets that Sensu uses, just like you would with any other secret. Sensu also respects Vault’s secret leases, working seamlessly to support those dynamic secrets and secret rotation workflows.

To get started, you’d take something like the below example, which shows a Sensu configuration with HashiCorp Vault secrets:

It’s a resource, like all other Sensu resources, so you can automate this when provisioning your Sensu clusters.

Demo

In the following demo, I illustrate how the Vault integration works, using a simple Docker Compose template that stands up a Sensu backend and Vault server in dev mode (note: this is only intended for testing and demo purposes, not production environments). Watch the demo, and follow along with the steps below!

Guide

The remainder of this blog post is a step-by-step guide to help you get started with secrets management for Sensu Go, and the HashiCorp Vault integration.

Prerequisites:

• Download & install Docker & docker-compose; see: https://docs.docker.com/compose/install/
• Download & install the Sensu CLI (sensuctl); see: https://sensu.io/downloads NOTE: we’ll run the Sensu Go server in a Docker container, but you’ll need the sensuctl CLI to interact with Sensu.
• Download & install the Vault CLI (vault); see: https://www.vaultproject.io/docs/install NOTE: we’ll run the Vault server in a Docker container, but you’ll need the vault CLI to interact with the Vault server.

Step 1: Start up the demo environment

$ git clone https://github.com/calebhailey/sensu-vault-demo.git 
$ cd sensu-vault-demo
$ sudo docker-compose up -d
Creating network "webinar_default" with the default driver
Creating volume "webinar_sensu-backend-data" with local driver
Pulling sensu-backend (sensu/sensu:5.20.1)...
5.20.1: Pulling from sensu/sensu
486039affc0a: Pull complete
d328af7475d0: Pull complete
cb728683b14d: Pull complete
6c0bd049f0a3: Pull complete
183ed819b4f9: Pull complete
2279a0c4f92f: Pull complete
4c06b482ee4b: Pull complete
Digest: sha256:f13625c8834dd293c8ee13b7f295266c4ec5e81bef58e2da2a6c156feb212c68
Status: Downloaded newer image for sensu/sensu:5.20.1
Creating webinar_vault_1         ... done
Creating webinar_sensu-backend_1 ... done

Step 2: Configure sensuctl to connect to the local backend

$ sensuctl configure -n --url http://127.0.0.1:8080 \
  --username sensu \
  --password sensu \
  --namespace default
$ sensuctl cluster health
=== Etcd Cluster ID: 3b0efc7b379f89be

      ID           Name     Error   Healthy  
 ────────────────── ───────── ─────── ───────── 

  8927110dc66458af   default           true

Step 3: Configure the Vault CLI to connect to the Vault server

$ export $(cat .env | grep VAULT_DEV_ROOT_TOKEN_ID)
$ echo "${VAULT_DEV_ROOT_TOKEN_ID}" > $HOME/.vault-token
$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.4.1
Cluster Name    vault-cluster-468b591e
Cluster ID      2d9aa556-a384-0b54-2523-346ae6e6d818
HA Enabled      false

Step 4: Create some Vault secrets!

NOTE: please rename slack-secrets.json.example to slack-secrets.json and update the contents of the file with a valid Slack webhook URL.

$ vault kv put secret/sensu/slack @vault/slack-secrets.json 
Key              Value
---              -----
created_time     2020-05-18T22:47:17.765882188Z
deletion_time    n/a
destroyed        false
version          1
$ vault kv get secret/sensu/slack 
====== Metadata ======
Key              Value
---              -----
created_time     2020-05-18T22:47:17.765882188Z
deletion_time    n/a
destroyed        false
version          1

======= Data =======
Key            Value
---            -----
channel        #demo
ttl            30s

webhook_url    

https://hooks.slack.com/services/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxxxxxxxxxxxx

Step 5: Configure the Sensu Vault provider & Slack handler

$ ll
$ sensuctl create -f sensu/vault.yaml
$ sensuctl create -f sensu/slack.yaml 
$ sensuctl handler list 
$ sensuctl secret list

Step 6: Create an event that should alert in Slack

$ sensuctl user create webinar --password $(uuid)
Created
$ sensuctl user add-group webinar cluster-admins
Updated
$ sensuctl api-key grant webinar 
Created: /api/core/v2/apikeys/ad3a7c47-6485-48af-a7e1-c9fc4a8a709e

Then, use the API key to create an event via the Sensu API.

NOTE: you’ll use the UUID portion of the last command output (i.e., the text that comes after /api/core/v2/apikeys/ ) to create your SENSU_API_KEY environment variable (as shown below).

$ export SENSU_API_KEY=ad3a7c47-6485-48af-a7e1-c9fc4a8a709e
$ curl -i -XPOST -H "Authorization: Key $SENSU_API_KEY" \
  -H "Content-Type: application/json" \
  -d
'{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"status":2,"output":"ERROR: failed to connect to database.","handlers":["slack","pagerduty"]}}' \
  http://127.0.0.1:8080/api/core/v2/namespaces/default/events

Step 7: Resolve the alert

$ curl -i -XPOST -H "Authorization: Key $SENSU_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"status":0,"output":"Database connection restored.","handlers":["slack","pagerduty"]}}' \
  http://127.0.0.1:8080/api/core/v2/namespaces/default/events

Need help? Check out the troubleshooting tips in my demo repo.

There you have it! We hope this post gave you an idea of the importance of secrets management — especially considering the challenges brought on by multi-cloud infrastructure — and offered a look into how secrets management can work with your monitoring. For further learning, check out the resources below, and join us on Discourse to stay updated on the goings-on in our community — including upcoming webinars and opportunities to learn from the monitoring community at large.

Resources:

• [Guide] How to use secrets management in Sensu Go
• Caleb’s Sensu + Vault demo on GitHub
• [Webinar recording] The Sensu + HashiCorp Vault integration
• Vault tutorial, from the HashiCorp learning library
• [Whitepaper] Unlocking the cloud operating model: security, from the HashiCorp website