Migrating to Sensu native silencing

Back in September we released Sensu Core version 0.26, adding silencing of clients and checks as a native construct in Sensu. In this post we aim to illuminate the specifics of this new approach to silencing and the possibilities it unlocks.

We call this new approach “native” because silencing of check results is now a concept built into Sensu Core itself. Prior to the release of Sensu Core 0.26, silencing check results — that is, preventing them from triggering a handler — was dependent on the handler’s implementation.

Sensu handlers written in Ruby using the sensu-plugin library automatically took advantage of that library’s functionality for querying the API for a list of stashes under the /stashes/silenced path. Handlers written in other languages, or otherwise without using the sensu-plugin library either implemented their own code to query the /stashes API, or couldn’t be silenced in the same way.

This approach has been a source of confusion and consternation for many, as attempting to use a simple unix command like mailx for handling messages can quickly turn into a yak-shaving exercise when silencing alerts from such a handler just doesn’t work in an intuitive way.

Even when using Ruby handlers with the sensu-plugin library in full effect, the /stashes implementation of silencing left something to be desired. For example, this approach does not make it possible to silence checks client subscription. In a highly-dynamic environment, new clients might be provisioned automatically, with client names not known in advance. Were this to happen during an incident, operators would have to silence new nodes as they come online to avoid alerting on incidends they have already otherwise silenced.

With the arrival of the /silenced API, scenarios like these become be a thing of the past.

The now-deprecated /stashes silencing implementation supports the following:

1. Silencing all checks on a specific client
2. Silencing a specific check on a specific client
3. Silencing a specific check on all clients

In comparison, the new native silencing implementation is more flexible, supporting:

1. Silencing all checks on a specific client
2. Silencing a specific check on a specific client
3. Silencing all checks on clients with a specific subscription
4. Silencing a specific check on clients with a specific subscription
5. Silencing all checks on all clients
6. Automatically removing silencing when a check returns to OK status (via expire_on_resolve)

For more detail on native silencing and the steps required to migrate from stash-based silencing, see our Silencing reference documentation.