Lesson 4: Introduction to Handlers

• Goals
• Handlers
  • Pipe Handlers
  • EXERCISE 1: Create a Handler that Sends Alerts to a Chat App
• Using Handlers to Store Observability Data
  • Events and Metrics
  • Handler Filters
  • EXERCISE 2: Create a Handler that Stores Metrics
• Discussion
  • Where Do Handlers Run?
  • Monitoring as Code and Sensu’s API-based Architecture
  • Sensu Plugin SDK, Templating, and Configuration Overrides
  • Additional Use Cases
• Learn more
• Next steps

Goals

In this lesson we will introduce Sensu Handlers, and show how to configure an alert handler and a metrics handler. This lesson is intended for operators of Sensu, and assumes you have set up a local workshop environment.

Handlers

Handlers are actions the Sensu backend executes on incoming observability events. Handler configurations are one of the most important building blocks within Sensu, because they determine what happens to events that flow through the Sensu pipeline.

Handlers can be used to send alerts, store observability data, create and resolve incidents, and trigger automated remediations. In the following exercises we will configure two handlers; one that send alerts to a chat app, and another that stores metrics to a time-series database.

Pipe Handlers

The most common type of handler is a pipe handler. A pipe handler can be any program that accepts event data as JSON input to STDIN, even a shell script. There are many existing handlers available in Bonsai.

Example pipe handler:

---
type: Handler
api_version: core/v2
metadata:
  name: example_pipe_handler
spec:
  type: pipe
  command: do_something.sh

EXERCISE 1: Create a Handler that Sends Alerts to a Chat App

Scenario

Your SRE team primarily communicates via a chat app like Slack or Mattermost. They want to recieve alerts as chat messages.

Solution

To accomplish this we will use the sensu-slack-handler. Because Mattermost is API-compatible with Slack, we can use the Slack handler with Mattermost. The handler will send event data to a channel in the workshop Mattermost instance.

Steps

1. Create a YAML File Containing the Handler Configuration

   Copy and paste the following contents to a file named mattermost.yaml:

   ---
   type: Handler
   api_version: core/v2
   metadata:
     name: mattermost
   spec:
     type: pipe
     command: >-
       sensu-slack-handler
       --channel "#alerts"
       --username SensuGo
       --description-template "{{ .Check.Output }}\n\n[namespace:{{.Entity.Namespace}}]"
       --webhook-url ${MATTERMOST_WEBHOOK_URL}
     runtime_assets:
     - sensu/sensu-slack-handler:1.4.0
     timeout: 10
     filters: []
     secrets:
     - name: MATTERMOST_WEBHOOK_URL
       secret: mattermost_webhook_url
   

   Understanding the YAML:

   • The asset identifier sensu/sensu-slack-handler:1.4.0 instructs the backend to download the handler executable from Bonsai.
   • The --description-template option uses a handler template to format the event into a message string.
   • The handler is configured using environment variables and secrets available to the Sensu backend.

2. Create the Handler Using the sensuctl create Command

   sensuctl create -f mattermost.yaml
   

   This command will create the handler configuration contained in the YAML file by sending it to the Sensu Handler API.

3. Verify that the handler was successfully created.

   sensuctl handler list
   

   Output:

     Name    Type   Timeout     Filters     Mutator            Execute            Environment Variables               Assets
    ─────── ────── ───────── ───────────── ───────── ─────────────────────────── ─────────────────────── ─────────────────────────────────
    mattermost   pipe     10   is_incident           RUN:  sensu-mattermost-handler                      sensu/sensu-slack-handler:1.4.0
   

   Do you see the mattermost handler in the output? If so, you’ve successfully created your first handler.

Using Handlers to Store Observability Data

Sensu is designed to be a pipeline for observability events, but does not store the events directly. If you want to keep a historical record of the data, a handler can be used. The handler converts incoming events into the required format, then sends it to a database for storage.

Events and Metrics

In Sensu, observability data is modelled as events. Some events contain metrics as part of their payload in the metrics property. Learn more about metrics in the metrics reference docs.

Handler Filters

Handlers can apply a filter to ensure that they only operate on matching events. There are some built-in filters available for common use cases, or you can write your own using a JavaScript-based Sensu Query Expression.

In the exercise below, we will use the built-in filter has_metrics to ensure that only events with a metrics property are processed by the handler.

EXERCISE 2: Create a Handler that Stores Metrics

Scenario

You want to store metrics in a data platform like Sumo Logic, Prometheus, or InfluxDB.

Solution

To accomplish this we will use the sensu-sumologic-handler. The Sumo Logic handlers send metrics to your Sumo Logic account.

Steps

1. Create a YAML file containing the handler configuration.

   Copy and paste the following contents to a file named sumologic.yaml:

   ---
   type: Handler
   api_version: core/v2
   metadata:
     name: sumologic-metrics
   spec:
     type: pipe
     command: >-
       sensu-sumologic-handler
       --send-metrics
       --source-host "{{ .Entity.Name }}"
       --source-name "{{ .Check.Name }}"
     runtime_assets:
     - sensu/sensu-sumologic-handler:0.2.0
     secrets:
     - name: SUMOLOGIC_URL
       secret: sumologic_url
     timeout: 10
     filters:
     - has_metrics
   

   Understanding the YAML:

   • The asset identifier sensu/sensu-sumologic-handler:0.2.0 instructs the backend to download the handler executable from Bonsai.
   • The built-in filter has_metrics is used to ensure that only events with metrics are processed by the handler.
   • The handler is configured using environment variables and secrets available to the Sensu backend.

2. Create the Handler Using the sensuctl create command.

   sensuctl create -f sumologic.yaml
   

3. Verify that the handler was created.

   sensuctl handler list
   

   Do you see the sumologic-metrics handler in the output? If so, you’ve successfully created the handler!

Discussion

In this lesson, we’ve only scratched the surface of what handlers can do. You learned how to create a handler using a YAML file, use handler templating to format event data, use built-in filters, reference assets published to Bonsai, and view a list of running handler configurations.

Where Do Handlers Run?

Handlers are part of the process stage of the observability pipeline. That means all of this happens on the Sensu backend, running the handlers in the same place where agents and checks send their observability data.

Monitoring as Code and Sensu’s API-based Architecture

Because Sensu is API-based, we were able to create the handlers remotely, using sensuctl to push the desired configuration to the backend via the Handler API.

We used a Monitoring as Code workflow, authoring the handler configurations with YAML files. We did not need to send any executable code, environment variables, or secrets along with this configuration. This means you can safely store the YAML configuration files in a git repo.

The executables are stored as assets in Bonsai (or a private asset server), and the secrets are stored in Vault. The Sensu backend will automatically download them as needed.

All of this works together to allow you to quickly add, remove, or change handler configurations in a live system at any time, without the need to redeploy.

Sensu Plugin SDK, Templating, and Configuration Overrides

Handlers developed using the Sensu Plugin SDK have built-in support for templating using the Golang text/template package. This can be used to merge observability data directly into the output, providing meaningful context and actionable alerts.

For example, an email handler could use an HTML message template that includes information like the status, number of occurences, or a customized playbook link.

Example: HTML email Template

<html>
The entity {{ .Entity.Name }} has a status of {{ .Check.State }}. The entity has reported the same status for {{ .Check.Occurrences }} preceding events.<br>
<br>
The playbook for responding to this incident is available at https://{{ .Entity.Labels.playbook_host | default "wiki.example.com" }}/observability/alerts/{{ .Check.Labels.playbook_id }}.
</html>

Another feature of the SDK is the ability to override handler configuration using metadata embedded in the event.

For example, you may want to send certain events to the #ops channel in Mattermost. The channel can be specifed in an annotation like sensu.io/plugins/slack/config/channel on a per-check basis.

Example: Check Configuration Using Annotations to Override Destination Channel

---
type: CheckConfig
api_version: core/v2
metadata:
  name: nginx-status
  annotations:
    sensu.io/plugins/slack/config/channel: "#ops"
spec:
  command: check-nginx-status.rb --url http://127.0.0.1:80/nginx_status
  publish: true
  subscriptions:
    - nginx
  interval: 30
  timeout: 10
  handlers:
    - slack

For more examples of configuration overrides using annotations, read the handler documentation for some frequently-used handlers (i.e. Slack, Pagerduty, and ServiceNow).

Additional Use Cases

With handlers, you can also send event data directly to a TCP or UDP socket, trigger automated remediations using tools like Rundeck or Ansible Tower, create and resolve incidents in Pagerduty, ServiceNow, or JIRA, and send alerts as Slack messages or emails.

For more complex workflows, multiple handlers can be stacked using handler sets.

Read more about handlers in the handler reference documentation.

Learn more

• [Documentation] “Sensu Handlers Overview” (docs.sensu.io)
• [Documentation] “Sensu Handlers Reference” (docs.sensu.io)
• [Documentation] “Guide: Send Pagerduty alerts with Sensu” (docs.sensu.io)
• [Documentation] “Guide: Send Slack alerts with Sensu” (docs.sensu.io)
• [Documentation] “Supported Handler Integrations” (docs.sensu.io)
• [Blog Post] “Reducing alert fatigue with GoAlert, Target’s on-call scheduling and notification platform” (sensu.io)

Next steps

Share your feedback on Lesson 04

Lesson 5: Introduction to Events