Sensu Go 6.5: Sensu Plus, High-performance Handlers, and Pipeline Resources

Sensu Go 6.5 is another feature-packed release and our first integration with the Sumo Logic Continuous Intelligence Platform. In this post, we’ll review the new features that make Sensu Go 6.5 such a banner release, and give you a sneak peek of what we have planned for future Sensu Go releases.

Monitoring as code meets machine data analytics

For Sensu users who want an integrated analytics engine to extract insights from Sensu’s observability pipeline, we are launching Sensu Plus – a bundled solution that includes Sensu plus the Sumo Logic Continuous Intelligence™ Platform, with simple node-based pricing. Sensu Plus is powered by a new built-in Sumo Logic integration now available in Sensu Go 6.5. Sumo Logic users who are adopting monitoring as code will also benefit from a number of related improvements in this release, as highlighted in this blog post.

Existing Sensu users are familiar with Sensu’s observability pipeline building blocks for collecting (Checks), filtering (Filters), transforming (Mutators), and processing (Handlers) of observability data. These components have been battle tested at scale and proven to be an effective solution for infrastructure and application monitoring in thousands of companies around the world. However, the process of configuring these components has been somewhat cumbersome, and in some cases difficult for new users to discover.

Introducing the Pipeline resource

Sensu Go 6.5 introduces the Pipeline resource to simplify configuration of the observability pipeline. Sensu Go Pipelines are composed of observability data processing workflows. Each workflow configures one or more Filters, a Mutator, and a Handler.

---
type: Pipeline
api_version: core/v2
metadata:
  name: sumologic
spec:
  workflows:
  - name: metrics
    filters:
    - api_version: core/v2
      type: EventFilter
      name: has_metrics
    handler:
      api_version: pipeline/v1
      type: SumoLogicMetricsHandler
      name: sumologic-metrics
  - name: events
    filters: []
    handler:
      api_version: core/v2
      type: Handler
      name: sumologic-logs

The original publish/subscribe mechanism in Sensu was an innovative solution that withstood the test of time, from public cloud to cloud native to multi-cloud infrastructure. By adopting a service-based model for monitoring configuration, new compute instances can “subscribe” to monitoring topics and begin reporting observability data in real-time. There is no need for brittle host-based monitoring configuration. The new Pipeline resource in Sensu Go 6.5 introduces a second publish/subscribe mechanism in Sensu.

The second benefit we get from Pipelines is that they open the door for new event processing capabilities (i.e. new types of handlers). Prior to this release, the Handler resource was responsible for both the configuration of a single processor (e.g. “send alerts to Pagerduty”), and the configuration of the workflow including which event filters and mutators should be applied. By separating these concerns we get a more intuitive UX, and we can now reference new types of handlers.

New streaming handler types

Sensu’s new streaming handler types, the Sumo Logic metrics handler and TCP stream handler, implement persistent connections for high throughput transmission of observability events to data platforms like Sumo Logic. Sensu’s core handler types – including pipe handlers, and TCP/UDP handlers – start a subprocess or open/close a new network connection for every individual event they process. This processing model is surprisingly resilient and very flexible for transactional event processing like alerts, incident management, and automated remediation jobs. But these core handler types are limited in their scalability. The new streaming handler types improve on this by distributing the workload across a persistent connection pool. In some workloads this yields substantial performance gains while also improving overall stability.

Support for JavaScript Mutators

New handler types aren’t the only major performance improvement in Sensu Go. We also added support for JavaScript Mutators, which are executed in the same embedded JavaScript runtime that we use for filters. Until now, Mutators in Sensu worked like pipe handlers – they shell out to a subprocess for each event they process. While this execution model is a useful primitive for extensibility – literally any program that can consume JSON via standard input, and emit string data via standard output can be used to transform Sensu events – it adds another processing bottleneck for high throughput use cases. The embedded JavaScript runtime provides a secure low-overhead sandbox environment for inline event enrichment and transformation, with substantial performance gains over subprocess execution.

Sensu Plus

All of the usability and performance improvements in Sensu Go 6.5 add up to a new packaged solution called Sensu Plus, which we announced at the inaugural Sumo Logic Illuminate event. With Sensu Plus, all Sensu users get turn-key integration with the entire Sumo Logic Observability Suite. All you need to get started is Sensu Go version 6.5.0 and a free Sumo Logic account. A new built-in Sensu Plus dashboard provides a high level overview of your observability data. The Sumo Logic App Catalog also features hundreds of dashboards for extracting key insights from metrics, log, and trace data collected by Sensu and/or a Sumo Logic collector. Check out the Sensu Plus pricing for more information, and see our Sensu Plus setup guide to get started.

Ready to upgrade to Sensu 6.5?

Sensu Go 6.5 is now available: download the latest version and review the release notes.

NOTE: To use the new Pipeline resource in Sensu Go 6.5, you must upgrade your Sensu agents to 6.5. Agents that are not upgraded to 6.5 will still run checks, send observability events to the backend, and use the handlers listed in check definitions, but they will not run pipelines.

Check out the full upgrade guide in our documentation.

Sneak peek

On the horizon for upcoming releases: the Sensu Catalog, a built-in browseable marketplace of Sensu integrations that can be enabled at the click of a button. Each integration in the Sensu Catalog is backed by a monitoring code template – a YAML file containing one complete solution and its various dependencies (e.g. runtime assets, secrets, etc). These templates include monitors for third-party services (i.e. Sensu Checks), and integrations with various systems of record (e.g. Pagerduty, ServiceNow, Sumo Logic, InfluxDB, etc). An open source repository containing over 100 of these templates is already under active development at https://github.com/sensu/catalog. At launch time the official Sensu Catalog will include over 200 solutions, including over 50 officially supported third-party integrations.

The Sensu Catalog will provide  everything new users need to get Sensu up and running, and rapidly deploy infrastructure and application monitoring solutions across an entire fleet of systems. Existing users will also benefit from the ability to develop custom catalogs – i.e. customizing the content available in the browseable marketplace.

We hope the changes we’ve described for Sensu Go 6.5 and the upcoming Sensu Catalogs solution will give you more options for monitoring your ever-changing infrastructure. Consider joining our community, where you can learn from the folks building Sensu and stay updated on the latest Sensu product releases.