Managing Sensu Go 6 with Puppet

The following is a guest post from Trey Dockendorf of Tailored Automation. He is a maintainer and the top contributor to the Sensu Puppet module. Trey has been developing Puppet modules for 10 years and specializes in system administration for high-performance computing.

Intro

The Sensu Puppet module has seen another major release up to 5.x to coincide with the release of Sensu Go 6.

The 5.0.0 bump for the Sensu Puppet module was a chance for us to introduce new features to better support Sensu Go 6, as well as remove code that had been deprecated and no longer needed.

The major changes in the 5.x release series of Sensu Puppet module include:

• Support for Sensu Go 6
• Support for Puppet 7
• Removing the need for old_password and old_agent_password parameters to manage passwords
• Removing deprecated properties of url, sha512 and filters for sensu_asset type, in favor of simpler builds hash.

Sensu Go 6 Support

One of the major changes with Sensu Go 6 was around how agent entities are managed. With Sensu Go 5 the agent entity was managed via the /etc/sensu/agent.yml configuration file. This file was used to do things like add subscriptions and assign labels or annotations to an agent entity. With Sensu Go 6, that agent.yml is used only for the initial bootstrap and subsequent updates to things like subscriptions, labels, and annotations is performed by making changes with sensuctl or the Sensu Go API.

As of Sensu Go 6.2, the agent subscriptions can be managed the same way they were in versions prior to Sensu Go 6.0. View our documentation to learn more.

The Sensu Puppet module now has the sensu_agent_entity_config native Puppet type that is responsible for managing the agent entity configurations via the Sensu Go API after initial bootstrap. There are helper defined types such as sensu::agent::subscription that wrap this new native type to make it easier to manage common agent entity configurations.

The /etc/sensu/agent.yml configuration file is still kept up-to-date with the agent entity state when using these helper defined types, so that an admin can look at a YAML file and see the desired state of the agent entity as well as re-bootstrap the agent entity if necessary.

Overall, the experience of using the Sensu Puppet module does not change much from Sensu Go 5 to Sensu Go 6; the biggest change was to the behinds-the-scenes code responsible for managing agents with Puppet. The Puppet module makes heavy use of the Sensu Go API to manage agent entities so that when things like subscriptions or labels are changed for an agent, those changes are made using the Sensu Go API and also written to agent.yml.

Using profiles to manage Sensu Go agents

One common design pattern with Puppet is the use of roles and profiles pattern. There are aspects of the Sensu Puppet module that were designed specifically to work well with profiles. The idea is that you have a profile for things like Apache or MySQL and the business logic of how to manage those services are included in a profile specific to that service. With the Sensu Puppet module, you can then associate agents with the necessary Sensu checks to properly monitor those critical services. This allows for fully automated deployment of services that includes automatic monitoring configuration as those services are deployed.

For example, let’s imagine that on your Sensu backend host you have checks defined like the following pseudo code:

sensu_check { 'check-cpu':
  ...
  subscriptions => ['linux'],
}
sensu_check { 'check-http':
  ...
  subscriptions => ['apache'],
}
sensu_check { 'check-mysql':
  ...
  subscriptions => ['mysql'],
}

With Sensu Go, you associate these checks to an agent by assigning that agent the subscription associated with the check. This association can be done using a profile class. Below are some pseudo code examples of possible profile classes that would be defined to manage the service and associate the necessary checks for those services.

class profile::linux {
  include crony
  include sudo

  sensu::agent::subscription { 'linux': }
}
class profile::apache {
  include apache

  sensu::agent::subscription { 'apache': }
}
class profile::mysql {
  include mysql::server

  sensu::agent::subscription { 'mysql': }
}
class profile::sensu::agent {
  class { 'sensu::agent':
    backends      => ['sensu-backend.example.com:8081'],
    subscriptions => ['base']
  }
}

If a host were to include all four of the above profiles in their role, that host would end up having the subscriptions ['base','linux','apache','mysql']. The values defines in multiple locations in Puppet would all be merged into /etc/sensu/agent.yml for the subscriptions value as well as make the necessary Sensu Go API calls to ensure these subscriptions remain assigned to the agent entity so that the proper checks are executed on that agent.

This same pattern can be used to define agent labels and annotations. The Puppet module provides the sensu::agent::label and sensu::agent::annotation defined types. The following examples illustrate using labels and annotations that get assigned to the agent entity via profile classes.

class profile::apache {
  include apache

  sensu::agent::label { 'contacts': value => 'web-admins@example.com' }
  sensu::agent::annotation { 'fatigue_check/occurrences': value => '2' }
}
class profile::mysql {
  include mysql::server

  sensu::agent::label { 'contacts': value => 'db-admins@example.com' }
}

If a host were to be assigned the Apache profile it would have the agent entity label contacts: 'web-admins@example.com and different host assigned the MySQL profile would have contacts: db-admins@example.com. Keep in mind that labels and annotations can only have string values, so declaring duplicate label or annotation keys will override prior values, unlike subscriptions which are merged together.

Conclusion

At this point, hopefully it is a bit clearer how to leverage Puppet to automate a Sensu Go deployment for seamless monitoring automation by leveraging both Sensu Go and Puppet. For further reading, check out the Sensu Puppet repo which includes many configuration examples for handlers like Slack, PagerDuty, and InfluxDB, and commercial features like LDAP authentication, PostgreSQL datastore, Vault secrets provider, and more.