Monitoring as code with Sensu Go and SensuFlow

Sensu creator and CTO Sean Porter recently wrote about “monitoring as code” and his perspectives on where the next generation of monitoring and observability workflows is headed. That post did a great job of outlining the concepts; this post will put theory into practice with SensuFlow, a new prescriptive monitoring as code workflow for Sensu Go, and its accompanying GitHub Action.

In a previous post, we shared how to integrate Sensu into your CI/CD pipeline with sensuctl prune. SensuFlow (a technical preview feature that has been publicly available since Sensu Go 5.19) takes it a step further — SensuFlow is a prescriptive “monitoring as code” workflow that uses the Sensu Go API and sensuctl CLI (including sensuctl prune) to synchronize monitoring code with Sensu deployments.

Note: The SensuFlow GitHub Action is now generally available in technical preview (currently version 0.4.0). The individual components are subject to change, but the overall workflow is here to stay. We recommend you start by testing SensuFlow (as code!) in a dev environment or in a test namespace in your current environment before promoting it to production.

Read on to learn more about SensuFlow, including how to get started.

Intro to SensuFlow

SensuFlow is a collection of operational best practices and tools, including the following components:

• A Sensu RBAC profile (service account) with sufficient privileges to manage all your monitoring configuration as code
• A labeling convention to designate which resources should be managed by this workflow
• A code repository containing Sensu monitoring code
• Integration with your CI/CD system that runs sensuctl commands using the aforementioned Sensu service account from the monitoring code repository

Let’s take a look at each of these components individually.

The repository of resource definitions can be any manner of source control repository (git, subversion, etc.). While a specific directory structure is not required for a monitoring as code implementation with Sensu Go, we will be suggesting one later in this post. Since sensuctl supports both JSON and YAML files, either can be used, but it is suggested that you use only one file type for consistency. For formatting and readability reasons, we recommend YAML. We also suggest you include all dependencies within a single template (YAML file). For example, if you have a handler that requires a dynamic runtime asset as well as a secret, the definitions for all three resources should be included in the handler template (YAML file).

The use of the Sensu API and CLI (sensuctl, which is effectively an API client) requires a Sensu user account with which to run commands. This user will need the appropriate RBAC permissions to manage the resources defined in your repository. If you are planning to manage namespaces using this workflow, these permissions need to be granted at the cluster level (cluster-role and cluster-role-binding). If the resources are confined to a single namespace, then the permissions can be granted within that namespace using a role and role-binding. This configuration (using a role and role-binding for namespace-specific resource management) requires Sensu Go 6.2.0 or newer — download the latest version.

The workflow makes use of the sensuctl prune command (and prune API) for removing resources that are no longer defined in your repository. The prune command supports removing resources either by username of the resource creator or by a defined label. We will be making use of labels in this workflow as they are more readily visible in the resource definitions themselves. They also ensure that prune is working in an explicitly defined manner.

Finally, you need a CI/CD system to pull this all together such that any updates to the resources in your repository are reflected in your Sensu configuration in a timely manner. This tooling can be anything of your choosing (e.g., GitHub Actions, Jenkins, GitLab CI, etc.); it simply needs to be able to check for changes in your repository and to run a script to execute the necessary sensuctl commands; or for more custom implementation, a script could make the corresponding API calls. Our example later will be based on a GitHub Action.

Now, let’s start to put these pieces together.

Setup

The following instructions represent our opinionated setup — your configurations will reflect your individual environment.

RBAC

To use SensuFlow, the first step is to create a user with an appropriate cluster-role and cluster-role-binding. This is the user that will be used to manage your Sensu resources using this workflow. Following the principle of least privilege, you will want to limit this user’s access to only those resources that you will manage with this workflow.

In our example configuration, the following resources will be managed using SensuFlow:

• Namespaces
• Roles
• Role bindings
• Assets
• Handlers
• Checks
• Hooks
• Filters
• Mutators
• Secrets

This list may not work for every environment. If you are working in a larger environment with multiple teams managing resources in disparate namespaces, your list may be limited to the more operational resources such as assets, checks, handlers, filters, mutators, and secrets. Your organization’s security policies may also affect the rights granted to any automated workflow.

For any resource to be managed with this workflow the user will need to have get, create, update, and delete access permissions for the corresponding API resources.

The user created will need a password assigned and these credentials will be needed by your CI/CD system to authenticate sensuctl to the Sensu API. Create the password hash for the password_hash attribute in the RBAC definition below using the following command:

sensuctl user has-password <password string>

Create a file with the following contents (including the substitution of the password_hash created above):

---
type: ClusterRole
api_version: core/v2
metadata:
  name: sensu_flow
spec:
  rules:
  - resources:
    - namespaces
    - roles
    - rolebindings
    - assets
    - handlers
    - checks
    - hooks
    - filters
    - mutators
    - secrets
    verbs:
    - get
    - create
    - update
    - delete
---
type: ClusterRoleBinding
api_version: core/v2
metadata:
  name: sensu_flow
spec:
  role_ref:
    type: ClusterRole
    name: sensu_flow
  subjects:
  - type: Group
    name: sensu_flow
---
type: User
api_version: core/v2
metadata:
  name: sensu_flow
spec:
  disabled: false
  username: sensu_flow
  password_hash: $2a$10$fOZaPTkZhEPVbwbXHY4LV.M8mv8yskRjL9zAtqVaLR8ppFb5vJHZq
  groups:
  - sensu_flow

Run the following command to create the above RBAC configuration:

sensuctl create -f </path/to/rbac-file.yaml>

Resource labeling

As mentioned above, we advocate the use of labels in your resource definitions. The following snippet of a Slack handler definition shows the label that will be used in our GitHub Action.

---
type: Handler
api_version: core/v2
metadata:
  name: slack
  labels:
    sensu.io/workflow: sensu_flow
[...]

This labeling is one of the key decision points when implementing SensuFlow since this is the basis for how the sensuctl prune and the prune API determine what resources are to be deleted if they are no longer in your repository. This not only affects the resources in your repository. If unmanaged resources (i.e., they are not managed as code) have this label, they risk being deleted as part of the workflow.

It is also important to be consistent across your organization with this labeling so as to prevent the spread of unmanaged resources or accidental deletion (as noted above).

Directory structure

The main driver behind using a prescribed directory structure is for maintainability and readability. Organizing resources in this way makes it easier for multiple teams to collaborate and reuse monitoring pipeline solutions originally developed by another team. Your team will need to find the solution that works best for your environment.

The directory structure we’ll use in our new GitHub Action (a monitoring as code reference implementation, shown below) includes a top-level namespaces.yaml file containing the Sensu resource definitions for the namespaces we will manage with the workflow. Alongside that we have a namespaces directory that contains all of the resources for each namespace.

namespaces.yaml
namespaces/<namespace>/checks/<checkname>.yaml
namespaces/<namespace>/handlersets/<setname>.yaml
namespaces/<namespace>/handlers/<handlername>.yaml
namespaces/<namespace>/filters
namespaces/<namespace>/mutators

Some things to note from the above structure:

• The prescriptive SensuFlow workflow is under active development, so the conventions are subject to change (see the GitHub repo for more information).
• Checks are configured to process events using handler sets so each handler set will need to be updated with the corresponding handlers that will process matching events.
• You will see we specifically do not have directories for assets, hooks, or secrets. This is because as part of our structure, any resource that requires an asset, hook, or secret will include those dependent definitions in their template (YAML file).

Namespace specification

One final note on the resource definitions prior to discussing our GitHub Action: Since our Action can manage multiple namespaces, our sensuctl commands reference those namespaces explicitly. This means that the resource definitions in our repository should not contain namespace attributes. If they do, they could potentially be created outside of the intended namespace. One side benefit of not including namespaces in the definitions and allowing the workflow to specify them is that it allows for easily replicating configurations by copying files between namespace directories.

The SensuFlow GitHub Action (CI/CD integration)

We are happy to announce the public preview release of the SensuFlow GitHub Action, a reference implementation of monitoring as code with Sensu Go and SensuFlow. You can use this GitHub Action to get started, or modify it for your own needs. The SensuFlow GitHub Action is MIT-licensed and the source code is available on GitHub. At the time of this writing, the SensuFlow GitHub Action is implemented as a Bash shell script that executes sensuctl commands in a Docker container, but you can use the script in your local dev environment or adapt it for your preferred CI/CD pipeline.

Before using this GitHub Action, you’ll need to manually configure the RBAC policy for the Sensu user meant to operate the sensu-flow action. The Action README provides reference RBAC and user account resources to help get you started.

A quick note for adapting the GitHub Action for other CI/CD platforms: the script driving the GitHub Action requires three dependencies: jq, yq, and sensuctl. When you are adapting the script for use outside of GitHub Actions, make sure you install those additional executables before running the script.

The Action makes use of several environment variables to customize how the script interacts with your environment. The minimum required variables for the Action are SENSU_BACKEND_URL, SENSU_USER, and SENSU_PASSWORD. All other environment variables are optional settings to help you tailor the SensuFlow experience.

The SensuFlow GitHub Action also provides some additional linting logic to ensure your resource definitions are self-consistent with respect to your chosen namespace directory structure and label matching regime used by SensuFlow. Beyond the basic sanity checking that the Action provides, you’ll also want to make sure your resources conform to the following general rules:

• Avoid using explicitly namespaced resources, as this will conflict with sensuctl’s global namespace argument taken from the directory naming structure of SensuFlow.

• Ensure all resources have the needed label and value used by sensuctl prune label matching.

NOTE: you may wish to star the GitHub Action repository and stay up to date with new releases as we’ll be adding a number of new built-in tests in future versions of SensuFlow.

And there you have it: a CI/CD workflow (in our example, using GitHub Actions) to ensure that any changes to your monitoring configuration are immediately populated throughout your Sensu deployment!

We want your feedback!

We’re really excited about this new workflow for managing your Sensu monitoring resources, but it’s still a new concept and we need your feedback to further enhance it. In particular, the SensuFlow GitHub Action makes use of the sensuctl prune command — also a technical preview — and could use some feedback from real-world users. If you’re interested in testing SensuFlow, but do not want to use it in production right away, the best thing you can do is to start with a dedicated Sensu namespace for SensuFlow automation to maintain. You can modify the reference RBAC and limit access for your SensuFlow user to just the testing namespace you’ve pre-defined. If you run into any problems or have ideas to further enhance the SensuFlow concept, please file issues against the Github Action repository.

Watch our webinar: Practical monitoring as code

Sign up to receive the recording of our on-demand webinar when it becomes available on February 25, 2021. You’ll learn:

• What monitoring as code means, and why it matters.
• How to get started with your own monitoring as code workflows.
• How the SensuFlow GitHub Action works.
• How to apply monitoring as code to your own CI/CD pipeline.