Learn Sensu: External services

Lesson 1: Handler annotations

Thanks for joining us! Over the next few lessons, you’ll learn all about using handlers to customize alerts, TCP and UDP handlers, secrets management, and mutators.

Before we get started, a couple caveats: for the purposes of this course, we’ll assume you’ve gone through our Learn Sensu Go course (or have that baseline knowledge, such as how to send Slack alerts) and have downloaded Sensu (which you’ll need to do to follow along). Finally, don’t forget to register your download so you can start earning rewards on your Sensu journey!

For the first lesson, we’ll go over how to use handler annotations to customize alerts. Annotations let you tune handlers on a per-entity or per-check basis, and you can use annotations to add data that’s meaningful to people or external tools that interact with Sensu.

For example, here’s how you’d tune a Slack handler channel for a specific check/entity (you’d either do this in the check’s metadata or in the entity’s agent.yml file):

annotations:
  sensu.io/plugins/slack/config/channel: '#monitoring'

Note that an entity annotation overrides the same-named check annotation.

Next lesson, we’ll dive into TCP/UDP handlers.

Lesson 2: TCP and UDP handlers

In this lesson, we’ll cover TCP and UDP handlers, which enable Sensu to forward event data to arbitrary TCP or UDP sockets for external services to consume. Learn more about TCP/UDP handlers in our docs.

You can use Sensu TCP/UDP handlers to connect directly with external services such as Splunk or Graphite. Here’s how:

type: Handler
api_version: core/v2
metadata:
  name: graphite_tcp
  namespace: default
spec:
  env_vars: []
  filters: []
  handlers: []
  mutator: only_check_output
  runtime_assets: []
  secrets:
  socket:
    host: localhost
    port: 2003
  timeout: 0
  type: tcp

There’s a built-in mutator that’s metrics-only, so you can send only the data you need (i.e., when sending metrics to Graphite using a TCP handler, Graphite expects that data to follow the Graphite plaintext protocol). To do this, simply add the only_check_output mutator in the handler configuration mutator string:

type: Handler
api_version: core/v2
metadata:
  name: graphite
  namespace: default
spec:
  mutator: only_check_output
  socket:
    host: 10.0.1.99
    port: 2003
  type: tcp

Next time, we’ll go over secrets management with environment variables.

Lesson 3: Secrets management

In this lesson, you’ll learn about secrets management and how to use the Sensu environments variable to make sure your secrets stay, well, secret.

As the name implies, Sensu’s secrets management eliminates the need to expose secrets in your Sensu configuration. When a resource definition requires a secret (like a username or password), Sensu allows you to obtain secrets from one or more external providers, like HashiCorp Vault. Learn more about secrets management in our documentation.

Secrets management works with checks, handlers, and mutators by using an environment variable. Here’s an example configuration of secrets that target a HashiCorp Vault:

type: Secret
api_version: secrets/v1
metadata:
  name: sensu-ansible
  namespace: default
spec:
  id: 'secret/database#password'
  provider: vault

As another example, here’s how you’d create a secret and update the Slack handler to use it for the webhook, using the env variable:

type: Secret
api_version: secrets/v1
metadata:
  name: sensu-slack-webhook-url
  namespace: default
spec:
  id: SLACK_WEBHOOK_URL
  provider: env

Then update your Slack handler with the following definition to reference that secret:

type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel ‘#monitoring’
  env_vars: null
filters:
runtime_assets:
- sensu-slack-handler
secrets:
- name: SLACK_WEBHOOK_URL
  secret: sensu-slack-webhook-url
type: pipe

So, why would you use secrets management to call out environment variables, as opposed to just using the environment variable as configured at startup? This surfaces the fact that the command for the resource in question makes use of an environment variable that is set at startup and it’ll make for an easy transition to Vault if you decide to do so in the future.

That’s all for this lesson! In our next and final lesson, we’ll dive into mutators and their use cases.

Lesson 4: Mutators

In this lesson, we’ll discuss mutators, which allow you to change the JSON representation of a Sensu event into a data format that your external service expects.

With TCP/UDP handlers, for example, mutators make it so you can change the data structure as needed for the service that you’re contacting. Mutators let you add additional information that’s not part of the Sensu event structure because you’re working just with the JSON representation, which is very flexible.

Handlers can specify a mutator to execute and transform event data before any handlers are applied; when the Sensu backend processes an event, it first checks the handler for the presence of a mutator, and will execute the mutator before the handler if it’s present.

Learn All About Mutators

Sensu also includes built-in mutators, which help you customize event pipelines for metrics and alerts.

Here’s a definition of a simple mutator that will add some metadata to an event based on the check that’s being run:

{ 
  "type": "Mutator", 
  "api_version": "core/v2", 
  "metadata": { 
    "name": "team-mutator", 
    "namespace": "default", 
    "labels": null, 
    "annotations": null 
  }, 
  "spec": { 
    "command": "jq 'if .check.metadata.name == \"http\" then .metadata.team = \"webOps\" else .metadata.team = \"Ops\" end'", 
    "timeout": 0, 
    "env_vars": [], 
    "runtime_assets": [] 
  }
}

Using the above definition, you can — for example — add a mutator to a debug handler to make sure your mutator is acting as expected.

Learn How

That’s all for this course (but you can always revisit the course content here! If you haven’t already, we invite you to sign up for our email course on checks, where you’ll learn about check definitions, including check tokens and hooks, proxy checks, check scheduling, and API keys. We hope you found the course useful, and either way, we’d love to hear what you think.