Secrets Management: Use Cases, Best Practices, and Tools

To provide proper visibility into the health and status of your systems, observability tools require access to the internal and external services you’re using, and Sensu is no different. In the past, this could mean exposing sensitive authentication credentials like usernames and passwords with local environment variables or even by including the secret information in your monitoring configuration. These practices are a security risk because they display your secret information to unauthorized users, making your observability tool into an attack point for nefarious actors.

Fortunately, you don’t have to choose between convenience and security with Sensu. Sensu’s enterprise secrets management feature allows you to keep your sensitive information accessible and private at the same time. Whether you choose Sensu’s built-in Env secrets provider or our integrated HashiCorp Vault or CyberArk Conjur provider, you’ll be able to confidently share your Sensu observability configurations without exposing sensitive information.

What Is Secrets Management?

Secrets management is a means of safely and securely storing, managing, and accessing authentication credentials (“secrets”) in your infrastructure. Secrets include credentials like usernames and passwords, access keys and secret keys, API tokens, and security certificates and keys. Secrets management means both the tools you use and the practices you follow to protect this sensitive information while making it available to authorized services like your observability tool.

Secrets Management Use Cases

The most obvious use case for secrets management is protecting the sensitive credentials your tools require. Secrets management ensures that your tools have the access they need without the security risk of exposing secrets by storing them in unencrypted plaintext.

For observability solutions like Sensu, secrets management facilitates self-service monitoring by making it easier and safer to share monitoring configurations. You can share your configuration files without worrying about exposing secret information, whether you’re sharing them for reuse within your organization or as examples in a public demo. Secrets management allows you to securely implement Sensu’s monitoring as code solution. Removing secret credentials from the underlying code and configuration also drastically reduces the time required to onboard new team members.

Secrets management also helps satisfy security and compliance requirements. Secrets are removed from the configuration and stored in a centralized location, separate from the code. Secrets management provides an audit trail to show which users and systems had access to different secrets at different points in time. Centralized secrets management allows you to automate secrets rotation or use dynamic secrets workflows so that you change secrets more often and improve security.

Secret Management Challenges

Failing to implement secrets management presents a number of challenges. First and foremost, your organization may have hundreds or thousands of secrets already stored throughout the codebase. Aside from the constant threat of exposure, keeping secrets in your code encourages other poor security practices like reusing the same secrets, using weak passwords, and failing to rotate or revoke secrets because doing so would require many time-consuming code updates.

Some organizations have decentralized control over authentication credentials, with each team managing their own secrets. Decentralized secrets management makes it difficult to audit security practices (or even maintain consistent security practices).

Lack of secrets management can result in inadequate or incomplete observability. You might be hesitant to deploy a robust monitoring system that covers every nook and cranny of your systems without secrets management in place to protect the array of credentials it requires. Not using secrets management also makes it difficult for DevOps teams to do their jobs, since they often need access to secret credentials for tasks like configuration management and orchestration.

Secrets Management Best Practices

Although the most appropriate processes and tools to use for secrets management will vary for different organizations, there are some best practices to observe when you’re setting up a secrets management strategy.

Never store secrets or any other sensitive information in unencrypted plaintext like code or configuration files. Instead, maintain secrets with a secure, centralized service that provides a single control plane for managing all secrets throughout your organization.

Set a secrets management policy that includes rules about the number and types of characters to use in secrets; sharing, reusing, and rotating secrets; and securely storing secrets. Don’t forget to describe how your organization will enforce the policy, whether that involves audits and log reviews, session monitoring, application settings, or other means.

Automate secrets management as much as possible to reduce the risk of human error. Manual processes leave room for poor security practices like sharing or reusing passwords and storing secrets separately from the organization’s centralized system. Automated secrets management also allows you to enforce expiration and rotation practices that help limit the damage malicious actors can do if they obtain secret information.

Make sure all of your organization’s third-party partners and vendors follow an appropriate secrets management program.

Implement a threat analysis process for detecting unauthorized access and secrets misuse. Your DevOps and Security teams should have processes in place for quickly responding to incidents, terminating sessions if needed, and remediating any resulting damage.

Secrets Management Tools

Secrets management tools allow organizations to centrally manage sensitive information like passwords, keys, tokens, certificates, and other credentials as well as securely transmit the sensitive information. They reduce the risk that secrets will be exposed in a data breach, disclosed to an unauthorized user, or stored inappropriately.

Many open-source and commercial secrets management tools are available, and many modern cloud computing platforms include built-in secrets management features as well. The Sensu Go commercial distribution includes several options for enterprise secrets management, including a built-in Env environment variable and dedicated secrets provider resources for CyberArk Conjur and HashiCorp Vault.

Secrets resources are portable across Sensu namespaces and clusters, so you can securely share your monitoring and observability configurations with other teams throughout your organization. You can also use Sensu’s role-based access control (RBAC) to configure access to your secrets resources and make sure that they’re only shared with the roles and users you specify.

When a Sensu check, handler, or mutator requires a credential like a password or access key, Sensu can fetch it from an external secrets provider and make it available to the Sensu resource with a temporary environment variable. Secrets are exposed only to Sensu services like environment variables—your secrets are not exposed in your Sensu configuration, and Sensu automatically redacts secrets from all logs, the API, and the web UI.

Let’s take a closer look at each of Sensu’s secrets management options. The examples demonstrate how to supply a PagerDuty Integration Key as a secret to a PagerDuty handler in Sensu.

Sensu’s Built-in Env Secrets Provider

The Env secrets provider exposes secrets from environment variables on your Sensu backend nodes. To use the Env secrets provider, first set your PagerDuty Integration Key as a backend environment variable. For example, in a RHEL environment:

echo 'SENSU_PAGERDUTY_KEY={YOUR_INTEGRATION_KEY}' | sudo tee -a /etc/sysconfig/sensu-backend

This configures the SENSU_PAGERDUTY_KEY environment variable to your PagerDuty Integration Key in the context of the sensu-backend process.

Then, create a Sensu secret that refers to the environment variable ID SENSU_PAGERDUTY_KEY. In this example, the secret is named pagerduty_key.

---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
spec:
  id: SENSU_PAGERDUTY_KEY
  provider: env

At the end of this section, I’ll demonstrate how to reference the pagerduty_key secret in handler commands.

CyberArk Conjur and Sensu

To use CyberArk Conjur as your external secrets provider, you’ll need to add an account for Sensu in Conjur, write and load a Conjur policy with the secret variable and host, and set a pagerDutyAPIKey secret that represents your PagerDuty Integration Key. After you complete the configuration in Conjur, you can create a secret and secret provider in Sensu.

The Sensu secret provider definition includes a map of your Conjur client attributes, like the account name, the HTTP or HTTPS endpoint CyberArk listens on, and authentication login:

---
type: CyberArkProvider
api_version: secrets/v1
metadata:
  name: conjur
spec:
  client:
    account: sensu.io
    appliance_url: http://localhost:8480
    login: host/Sensu/sensuBackend
    api_key: <API_KEY>
    timeout: 1s
    ttl: 60s

Next, create a corresponding secret for the conjur secret provider that references the pagerDutyAPIKey secret you configured in Conjur:

---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
spec:
  provider: conjur
  id: Sensu/pagerDutyAPIKey

Read on to see how to use the pagerduty_key secret in handler commands.

HashiCorp Vault and Sensu

Using HashiCorp Vault as your external secrets provider also requires some setup on Vault before you can create a Sensu secrets provider and secret. Specifically, you must authenticate via the HashiCorp Vault integration’s token method or transport layer security (TLS) certificate method.

Once you’re authenticated in Vault, write your PagerDuty Integration Key as a secret into Vault:

vault kv put secret/pagerduty key={YOUR_INTEGRATION_KEY}

Now you can create your secrets provider in Sensu. If you used the token authentication method, the secrets provider definition should look similar to this:

---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: http://localhost:8200
    token: <root_token>
    version: v2
    tls: null
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100

If you used Vault TLS certificate authentication, the secrets provider definition will look like this:

---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: https://vault.example.com:8200
    version: v2
    tls:
      ca_cert: /path/to/your/ca.pem
      client_cert: /etc/sensu/ssl/sensu-backend-vault.pem
      client_key: /etc/sensu/ssl/sensu-backend-vault-key.pem
      cname: sensu-backend.example.com
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100

Then, create the secret resource in Sensu:

---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
spec:
  id: secret/pagerduty#key
  provider: vault

Now you can securely pass your PagerDuty Integration Key in a Sensu handler.

Use Your Secret in a Sensu Handler

With your pagerduty_key secret configured, whether with Env, CyberArk Conjur, or HashiCorp Vault, you can securely pass your PagerDuty Integration Key in Sensu checks, handlers, and mutators. Here’s how that would look in a Sensu handler definition:

---
api_version: core/v2
type: Handler
metadata:
  name: pagerduty
spec:
  type: pipe
  command: pagerduty-handler --token $PD_TOKEN
  secrets:
  - name: PD_TOKEN
    secret: pagerduty_key
  runtime_assets:
  - pagerduty-handler
  timeout: 10

The handler’s secrets array defines the name/secret pairs to use with command execution. In this example, your pagerduty_key secret is named PD_TOKEN. The handler references $PD_TOKEN in the command, which requires your backend to request the pagerduty_key secret from your secrets provider.

What’s Next?

Now that you know more about Sensu’s secrets management features, you can use it to protect your sensitive information and keep your Sensu configuration secure.

Start by downloading and installing Sensu. Then, follow our guide to configure secrets management for your Sensu instance and make Sensu a partner in your secrets management strategy.