Sensu Integration Catalog: Engineering an Open Marketplace

ICYMI, the recent release of Sensu 6.7.0 introduced the Sensu Integration Catalog – an open marketplace for Sensu Go. Along with the release, we also hosted a webinar highlighting how the Sensu Integration Catalog unlocks self-service infrastructure monitoring. In the webinar, we talked about the “what and why?” – what problems does the Catalog solve, and why we decided to solve them with a marketplace. In this blog post, I’m going to be doing a technical deep dive into the “how?” – how it works, and how to get started developing your own integrations. If you’re interested in these details of what makes the Catalog integrations tick, keep reading.

The story so far…

We’ve been experimenting for over a year now on how to lower the barrier to entry for new Sensu users by providing curated solution templates. The Sensu Integration Catalog is the result of those efforts.

Some of our more active community members may have seen us working on the catalog content for the past few months, focusing on sharable resource templates for pre-defined monitoring workloads. We were able to get most of the way there with static YAML templates, but these ultimately required users to be comfortable with the Sensu APIs, CLI, and monitoring as code workflow. What we wanted was an even easier way to help new users get started without having to learn so much about how Sensu works. To accomplish this, you would need to customize the templates to your environment (e.g. DNS configuration and/or secrets) and integrate Sensu with the other tools already in use in your organization. The Sensu Integrations Catalog is exactly the user experience we’ve been looking for.

Putting the Pieces Together

There are 3 pieces that make the Integrations marketplace experience work.

1. Open-source integrations (https://github.com/sensu/catalog).

   This repository is the most important component of the Sensu Integration Catalog – it contains the curated integration content, in the form of Sensu resource definitions and integration definitions which define how they are configured via prompts in the web app. This is the primary focus of this blog post so we’ll take a closer look at integrations below.

2. A static API generator tool (https://github.com/sensu/catalog-api).

   The catalog-api tool generates a static catalog API based on a local checkout of an integrations repository (i.e. the sensu/catalog repo). We’ll talk more about the catalog-api in a future blog post, where we’ll demonstrate how to host a customized private catalog behind your organization’s firewall.

3. The in-app marketplace.

   The Sensu Integration Catalog is built directly into the Sensu Go web app. This in-app marketplace consumes data from a Catalog API such as https://catalog.sensu.io, the official API (which is also produced by the catalog-api tool). Most users will interact with this component when they are getting started with Sensu.

Tip: Check out the Sensu Go Workshop to start evaluating the Sensu Integration Catalog in your own dev environment.

The Sensu Integration Repository

You can find the official Sensu Integration repository here: https://github.com/sensu/catalog.

Sensu Integrations are a new way of curating monitoring solutions for Sensu, combining plugins with simple web-based user prompts. If you are a Sensu operator, this is where you can contribute your configuration experience to help other Sensu operators. Integrations are designed to be a low-code way to capture the tribal knowledge of operators, creating shareable solutions that can be deployed with the click of a button.

The integration repository content is fully open source. 100% of the official Sensu Integration Catalog contents — including commercially supported and community supported integrations — are available on GitHub for users to clone, fork, and contribute to. Adding an integration is easy thanks to Sensu’s monitoring-as-code underpinnings (every integration starts with a valid Sensu check or pipeline). New integrations with customized user prompts can be added with as little as two YAML files and a README.

Integrations are NOT plugins

It is super important to note: Integrations are a new concept for Sensu and are curated monitoring solutions. Sensu plugins still exist and power many of the integrations you’ll find in the catalog. But the key difference between the two concepts is that integrations include all the necessary resource configuration, while plugins are just the executables that you reference when building Sensu resources. Plugins do the bulk of the work behind the scenes, assets are a convenient way to package and distribute those plugins, and now integrations let you describe and share a customizable set of resource configurations that represent a complete monitoring workload.

The anatomy of an integration

The catalog repository is organized following a simple naming convention. Individual integrations are grouped into subdirectories named after a service or resource. For example, the slack-alerts integration is located under the path integrations/slack/slack-alerts in the catalog repository.

Each integration directory contains five files:

• sensu-resources.yaml

  This file contains one or more Sensu resource definitions. In some cases, a sensu-resources.yaml template may be installable but non-functional because they are missing environment-specific customizations (e.g. DNS configuration or secrets). The intent is to provide a clean version of the resource definitions devoid of any integration marketplace metadata so they can be used outside of the marketplace. They also serve as a reference for users who need to manage an internal marketplace with different or proprietary customizations.

  NOTE: If you’re a Sensu admin inside a large organization supporting multiple product teams, you might prefer to maintain  your own in-house marketplace, curated to only include integrations for  approved third-party systems and referencing privately hosted assets. In fact, we designed the catalog to make that possible. We’ll cover how to do that in more detail in a different blog post. Internally, we actually run our own private marketplaces when developing new integrations so we can do a full end-to-end test locally before submitting pull requests to the official catalog repository.

• sensu-integration.yaml

  This file defines the integration metadata (e.g. name, tags, etc), configuration prompts, and the associated resource patching rules used to customize the templated resources. This file describes a catalog/v1.Integration resource that can be consumed by the catalog-api tool. The structure of this resource should look familiar to experienced Sensu users as the Integration resource definition is patterned similarly to Sensu API resources (though the Sensu API does not consume these directly).

  NOTE: The prompting system is actually really slick. The integration resource defines its own internal user variable substitution scheme (using different syntax than Sensu token substitution to avoid conflicts). It supports multiple data types – simple types as well as composites like arrays and objects. It also supports string interpolation, and you can even use integration user variables to patch token substitution strings used in check command definition. You can check out how the integration variable system works here: https://github.com/sensu/catalog#integration-api-specification. Details about prompt and patching can be found at the bottom of the api specification section.

• README.md

  This contains information about the integration in markdown format. This describes a high-level overview of what the integration will do so users can review it before installing. The in-app marketplace displays the contents of the README as the homepage for the integration.

• CHANGELOG.md

  A log of the integration’s change history. Useful to note when the prompting logic was updated, or when included asset versions changed etc.

• logo.png

  Optional file to use in the catalog marketplace display.

A practical integration walkthrough

Let’s take the slack-alerts integration from the official repository as a working example. Starting from the root of the repository checkout, you can find it at integrations/slack/slack-alerts/.

The sensu-resources.yaml provides a templated version of the Slack handler, a pipeline with a couple of built-in filters applied, and an asset resource referring to the slack handler asset hosted on Bonsai. You could install these directly using sensuctl create -f sensu-resources.yaml, but  some additional configuration is required before the handler can send alerts to Slack.

Take a look at the handler definition in the sensu-resources.yaml file:

A SLACK_CHANNEL environment variable is referenced, but not defined. In addition, the plugin requires a webhook url. That should be considered sensitive information, so it is left out of the resource definition, and will be provided by the prompting system.

This is where less experienced Sensu users can stumble. Even when given a resource definition from another user’s environment, you may not know know what needs to be reconfigured. This requires some guidance – a conversation between you and the person providing the resource definition  – before you install the Sensu resources yourself. The catalog lets us encode that guidance into a series of input prompts. Let’s look at the slack-alerts integration as an example of what I’m talking about.

The prompts section describes the inputs presented to the user to guide them through a series of questions, and the user provides values for each. To customize this particular handler you need to answer two questions: what webhook URL do you want to use and what Slack channel to send the alerts to. We can’t encode these into the sensu-resources.yaml template directly, because they will be different for every user, but we can gather the necessary information using prompts, then patch the resources file before install.

Let’s look at the channel prompt in detail.

The value of a Slack channel is a string, prepopulated with a default value of #sensu. It’s marked as required, meaning the catalog won’t let you install the integration if you leave the channel blank.

As you can see, it is very easy to describe this declaratively in YAML. This sort of string value prompt will suffice for a vast majority of the customization needs, but integration prompt system also allows for more complicated things like pull-down lists and can even populate suggestions from the Sensu backend (great for configuring which subscriptions a check should be added to)!

Okay, so that’s how we will ask the user for the Slack channel, but the sensu-integrations.yaml also needs to define how to patch the sensu-resources.yaml with the user input. Here’s the relevant patch section:

This patch operation tells the catalog to take the value of the slack_channel prompt we just defined and use it as part of a new environment variable statement, added to the Slack handler resource in sensu-resources.yaml. The other patches involve adding the webhook URL, either as a secret reference or as an environment variable, depending on how the user answers the webhook URL prompt.

Prompt and patch, it’s like having a conversation with the original integration author to make sure you do the necessary customizations to the resource configuration for your environment. This approach solves a long standing problem with communicating operator wisdom between Sensu community members. The prompt and patch syntax offers a simple way to have that conversation as part of the install process. With just a few lines of YAML, we can describe both required and optional customizations,giving less experienced users a guided path to get new monitoring workloads up and running in their environment.

And this is just the start! One of the hardest parts of transferring operator wisdom has always been how to handle sensitive information – like the Slack webhook URL. We can tackle this with catalog integrations too! Take a look at the prompt for the slack_webhook_url. It’s not as simple as the slack_channel question, because we have to guide users through a more nuanced set of choices with regard to where sensitive data is stored, but it is possible to do in this framework.

Sensitive information like secrets are some of the hardest things to talk someone through in a community discussion, even in synchronous discussion channels. No experienced user wants to see someone they are helping accidentally leak sensitive info, when they were just intending to get validation from an expert on the edits they were making. With the catalog integration system,we now have a way to express this dialog, and be much more confident that the correct resource attributes are patched, reducing possible human errors.  We’re working on simplifying this even further, requiring even less YAML to describe prompts for secrets. It’s a difficult problem, because we want to encourage users to handle sensitive information in a secure manner. But we’re confident we can iterate on the catalog-api and simplify secrets customization even further. Stay tuned, definitely more to come on that!

Wrap Up

I hope you enjoyed this high-level introduction to the implementation details behind the Sensu Integration Catalog. There’s a lot more under the hood than we can cover in a single blog post, so stay tuned for Part 2. In the interim, please take a look around the official integration catalog repository, and give the integration install process a try. You can use the Sensu Workshop’s containerized environment to spin up a testing environment for Sensu in Docker (or Podman) where you can test drive the integrations. In the next blog post, we’ll do a deep dive into the developer tooling, and walk through the process of developing an integration and submitting it to the open source integration catalog.