Sensu Integration Catalog: One Static API to Rule Them All

ICYMI: in Part 1 of this blog post, I introduced Sensu Catalog Integrations, one of the three components that make the Sensu marketplace work.

In this post, I want to cover the second piece, the Catalog API generator. This is the tool that consumes the github.com/sensu/catalog repository content and renders static http API content the Sensu web app can consume.

The Catalog API Generator

The official Sensu Integration Catalog API hosted at https://catalog.sensu.io is powered by the catalog-api – an open-source static API generator. The catalog-api is a CLI tool that converts low-code Integration definitions (e.g. the contents of github.com/sensu/catalog repo) into static API content that can be hosted on any HTTP web service. The official Catalog API production environment is hosted in an AWS S3 bucket, with AWS CloudFront as a global CDN provider, but catalog-api output is compatible with any HTTP server (e.g. NGINX or Caddy). The catalog-api CLI tool supports reproducible builds, and you can use it to generate your own private catalog API. You can find the catalog-api tool at https://github.com/sensu/catalog-api. It’s written in the Go programming language, so a quick git clone & go build will get you a local command you can use, and then it’s just a matter of pointing it at a directory containing integrations.

Here’s how easy it is to generate a local catalog api from the Sensu Integration Catalog:

$ git clone https://github.com/sensu/catalog-api
$ git clone https://github.com/sensu/catalog
$ cd catalog-api
$ go build
$ cd ../catalog
$ ../catalog-api/catalog-api catalog generate

The catalog-api tool’s catalog generate command will generate the static api into a temp directory.

Support for Private Catalog APIs

Sensu administrators take note… I said private catalog APIs. I know there are a lot of you out there working inside organizations that are going to want/need to offer your own curations of Sensu Integrations for teams making use of your private asset mirror inside of your org’s infrastructure.  The catalog-api tool is designed with that use case in mind.  We’d love for you to share as much as you can with the wider community in the public catalog, but we understand the day to day practicalities means you’ll need to host a modified set of curations derived from the public catalog, including some things that don’t make sense to share. We can’t directly help you build the private content, but we can make sure you have the exact same tools we use to build your private catalog API that we use to build the public one. I won’t get into the details of setting up a private catalog here (I know I’m a little bit of a tease.) We’re working on separate documentation for setting up a private catalog, but for the adventurous, you can review the GitHub Action workflows in the Sensu Integration Catalog repository to see how the catalog-api tool is used to generate the official Catalog API.

What I want to do here is show how to use the catalog-api tool to generate a local catalog API to develop new integrations. Whether you’re contributing to the official Sensu Catalog or building a private catalog, you’ll probably need to use a similar workflow to develop integration content.

Local Development Environments

Not only can the catalog-api tool generate a static api, but it also offers a built-in web server for use as a development environment. You can run the command: catalog-api catalog server and get a local catalog api server you can connect to for local development. This server is not appropriate for production use, but it’s great for developing new integrations.

How it works

We love RESTful HTTP JSON APIs. We decided early on that the “v1” Catalog API would be read-only, and without the need for handling PUT/POST/DELETE requests, we realized we could build the Sensu Catalog around a static API Generator that could be hosted with minimal operational overhead.

The catalog-api tool consumes integration content matching the integration spec in https://github.com/sensu/catalog. The catalog-api tool implements a reproducible build strategy that generates builds into a checksum-based output directory structure. The path to the “latest” or “production” Catalog API content is managed by a version.json file (literally https://catalog.sensu.io/version.json), which instructs the Sensu web app to load API contents from a given checksum subdirectory.

Here’s an example of the version.json I just created using the catalog-api catalog generate command:

{"release_sha256":
 "5029648381dff2426ea247147456b4f1227fd6d9050fa42f0660e67a218f8c87",
"last_updated":
 1655840571
}

The release_sha256 points to a subdirectory derived from the checksum of the contents. Here’s the contents of my locally generated api directory:

$ tree /tmp/generated-api/ -L 2
/tmp/generated-api
├── release
│   ├── 5029648381dff2426ea247147456b4f1227fd6d9050fa42f0660e67a218f8c87
│   └── version.json

The idea here is if you run catalog generate multiple times you get the exact same release directory. But if you make any changes you’ll generate a different directory and you can store these side-by-side in a production environment (such as an S3 bucket). If you need to roll back to an older build, all you have to do is change the value of the release_sha256 in this file to point to one of the other release directories.

The catalog-api tool also consumes git tags, parsing integration-specific git tags to manage and generate versioned integrations. This makes it possible to give users access to old versions of integrations to hedge against regressions in individual integrations.   For example, in the Sensu Integration Catalog repository, we have a couple of ansible-tower-remediation version tags defined:

$ git tag --list |grep ansible-tower-remediation
ansible/ansible-tower-remediation/20220223.0.0
ansible/ansible-tower-remediation/20220421.0.0

And here is what the catalog-api tool generates using these tags:

$ tree /tmp/generated-api/ -L 7
/tmp/generated-api/
├── release
│   ├── 5029648381dff2426ea247147456b4f1227fd6d9050fa42f0660e67a218f8c87
│   │   └── v1
│   │ ├── ansible
│   │ │   ├── ansible-tower-remediation
│   │ │   │   ├── 20220223.0.0
│   │ │   │   │   ├── CHANGELOG.md
│   │ │   │   │   ├── img
│   │ │   │   │   ├── logo.png
│   │ │   │   │   ├── README.md
│   │ │   │   │   └── sensu-resources.json
│   │ │   │   ├── 20220223.0.0.json
│   │ │   │   ├── 20220421.0.0
│   │ │   │   │   ├── CHANGELOG.md
│   │ │   │   │   ├── img
│   │ │   │   │   ├── logo.png
│   │ │   │   │   ├── README.md
│   │ │   │   │   └── sensu-resources.json
│   │ │   │   ├── 20220421.0.0.json
│   │ │   │   └── versions.json
│   │ │   └── ansible-tower-remediation.json

Finally, the catalog-api CLI provides two additional subcommands that aid in the Integration development workflow: a built-in server for local development (catalog-api catalog server ), and a validation subcommand ( catalog-api catalog validate) for CI/CD.

When doing local development of a new integration I use the catalog-api catalog server command and reconfigured the Sensu Workshop environment to point to my local catalog api server url.

Want to see how I use the catalog-api tool in my integration development workflow? Take a look at these videos, where I develop an entirely new AWS RDS integration from scratch. In Part 1 of the video, I develop the AWS RDS reference Check resource that works for my environment. In Part 2 of the video, I adapt that check into a reusable Marketplace Catalog integration. I make use of the catalog-api tool to spin up a local catalog API server and point my Sensu workshop environment at it to test the integration prompts for the new integration as I’m developing it.

Wrap up

I hope you’ve enjoyed part two of our look behind the scenes of the Sensu Integration Catalog. In Part 1, we provided an overview of the new Sensu Catalog Integration specification – one of three components that make up the new integration marketplace offering, and the content that is used to populate the in-app marketplace. In this blog post, we covered the novel API generator CLI tool (catalog-api), and local development environment setup for Catalog contributors. Stay tuned for Part 3, where we’ll walk through the third component of the Sensu Integration Catalog: the in-app marketplace browser. In the meantime, give the Catalog a try by downloading the latest release of Sensu Go and installing some integrations. Then download the catalog-api tool and setup your local development environment to contribute enhancements and/or new integrations or even add your own Sensu workflows as community supported integrations for all Sensu users!