Integrating Sensu Go into your CI/CD pipeline with sensuctl prune

Since the release of Sensu Go, many in our community have told us Sensu is easier and faster to deploy, more portable, and more compatible with containerized and ephemeral environments (as compared to Sensu Core, the original version of Sensu).

In a recent webinar, I talked about integrating Sensu Go with your CI/CD pipeline and how to use the sensuctl prune command to keep your Sensu resources in a declarative state, reducing dependence on traditional configuration management tools. In this post, I’ll take you through the steps I demonstrated during the webinar, and also include the videos of the demos.

One quick note: prune is an alpha feature introduced in Sensu Go 5.19, and as such, it’s subject to change. We suggest you start by testing it in a dev environment or in a test namespace in your current environment.

Easier operations with sensuctl prune

sensuctl prune allows you to safely and easily remove resources you no longer need. Using sensuctl prune as part of your CI/CD workflow means you can put more of your configurations into a declarative repository, including public repos, without sacrificing security (especially when used in conjunction with Sensu’s secret management features).

You can have your Sensu resources just as they’ll be written and used, and you don’t have to get into using any special languages for templating just so you can interface with a configuration management tool. Instead, you can have your config management push out the sensuctl command to prune your resources and put them back into line with that set of resources as they exist in the repo.

What does sensuctl prune do?

The prune command deletes resources that don’t match the current resource definition in your version control system. Any updates to existing resources, and additions of new ones — such as checks or handlers — are automatically handled with sensuctl create commands.

The prune command has two use cases, username and label. The username use case keys off the user who creates a resource, and label makes use of a custom-defined per-resource label. This lets you mix and match your CI/CD jobs with different prune and create commands to create your resources and have them managed by different jobs.

sensuctl prune in action

The demo below will show you how integrating sensuctl into the CI/CD pipeline takes care of removing and managing resources. For the purposes of this demo, I am using Jenkins, one of the popular CI/CD tools. But you’ll see that my build steps are really just a few simple shell commands that can be replicated In any CI/CD tool you might favor. Sensu Go can be integrated with any of the popular CI/CD tools. We took a poll during the webinar and found that while many of our webinar participants were using Jenkins, Travis, or GitHub Actions, a fair share of folks were using GitLab.

Here are the steps, and if you want to watch the demo, the video is below.

First I’ll show you how my environment is set up. I’m logged in as admin and referencing my prune-demo namespace. I don’t have any resources defined, and instead of typing out a bunch of sensuctl check list and sensuctl handler list commands to demonstrate this, I’m going to shortcut and use the sensuctl dump command.

sensuctl dump checks,assets,handlers,filters,mutators,secrets/v1.Secret | wc -l

In my current directory I have two example check configurations, so I create a CPU check using one of them.

sensuctl create -f check-cpu.yaml

If I look at that resource now:

sensuctl check info cpu --format yaml

One of the things to pay attention to here is the created_by: user. As I showed earlier, I’m logged in as admin here. So metadata gets added, saying this was created by admin.

metadata: 
  annotations:
    fatigue_check/occurrences: "3" 
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: cpu
  namespace: prune-demo

The other thing to pay attention to is this label that says managed by: sensuctl. This is the key to the sensuctl prune command: It will not prune resources created outside of sensuctl. If you create a resource by using the API directly, and you don’t insert this label, prune will not touch that resource. The pruning action is based off of the label and the piece of metadata saying which user created it.

Now I’m going to prune checks in a dry-run mode. I point sensuctl prune at my resource definition, which is what I expect my resources to be — what to have prune compare against. The dry run comes back and says my resource definitions match, so I have nothing to prune, as indicated by the empty set as an output.

sensuctl prune checks -d -f check-cpu.yaml

[] 

If I were to point my prune command at that other check resource definition, though, it’s going to say it doesn’t match what’s in that resource definition. So sensuctl will prune it, deleting this CPU check because it doesn’t match my resource definition.

sensuctl prune checks -d -f check-disk.yaml
[
  {
    "type": "CheckConfig",
    "api_version": "core/v2",
    "namespace": "prune-demo",
    "labels": {
      "sensu.io/managed_by": "sensuctl"
    },
    "annotations": {
      "fatigue_check/occurrences": "3"
    },
    "created_by": "admin"
  }
]

Note that this is a simplistic example, because I’m just pointing to single resource definitions.

Now I’m going to switch to the pipeline user I’ve defined, and I’m going to run the same prune command from earlier. And it’s going to report back…nothing. That’s because the sensuctl prune command assumes by default that the user you’re running as is the user you want to manage resources for.

If I change the sensuctl prune command to include --users admin, then sensuctl prune will actually delete the resource, because it is owned by admin.

Those are the simplest examples. Now we’re going to put this to work inside a CI/CD pipeline.

Here in my Jenkins jobs list, I have two jobs defined:username and labels.

We will take a look at the username job first. It’s polling my demo GitHub repository every five minutes for changes.

We’re using a built-in secret for the pipeline users, for using sensuctl. So we’ll create resources here as the pipeline user.

The build job itself is nothing more than a shell script (shown below), rather than as a screenshot from Jenkins.

# ${SENSUCTL_USER} and ${SENSUCTL_PASSWORD} are secrets in Jenkins
test -d /var/lib/jenkins/sensuctl/${SENSUCTL_USER} || mkdir -p /var/lib/jenkins/sensuctl/${SENSUCTL_USER}

SENSUCTL_CONFIG_DIR=/var/lib/jenkins/sensuctl/${SENSUCTL_USER}
sensuctl configure -n --username ${SENSUCTL_USER} --password ${SENSUCTL_PASSWORD} --url https://backend:8080 --namespace prune-demo --config-dir ${SENSUCTL_CONFIG_DIR}

TMPFILE=resources_$$.yaml

# prune only works currently with a single file, so create it from
# the files in the repo
if test -d username
then
  find username -type f -name '*.yaml' | xargs cat >> ${TMPFILE}
  sensuctl prune checks,handlers,filters,mutators,assets,secrets/v1.Secret --users ${SENSUCTL_USER} -f ${TMPFILE} --config-dir ${SENSUCTL_CONFIG_DIR}
  rm -f ${TMPFILE}
  # Only try to do a create if there are the appropriate files in the direcotory
  if find username -type f | grep -q -i '.yaml'
    then
    echo "Running sensuctl create"
    sensuctl create -r -f username --config-dir ${SENSUCTL_CONFIG_DIR}
  fi
else
  sensuctl prune checks,handlers,filters,mutators,assets,secrets/v1.Secret 
    --users ${SENSUCTL_USER} -f .nonexistent.yaml --config-dir ${SENSUCTL_CONFIG_DIR}
fi

First it configures sensuctl to run commands in the prune-demo namespace using the provided Jenkins secrets for the pipeline user.

Then it looks for a directory in my Git repo called username because I have my Git repo split into labels and usernames for the two different build jobs in this demo.

It should be noted that there’s one current issue with sensuctl prune: It works only against a single file of resource definitions, not a directory. We’re working around that in these examples by doing a find in that directory, finding all the yaml files and concatenating them together.

From there, we run the prune command against that concatenated resources file. We’ll prune any checks created by SENSUCTL_USER (pipeline in this case) that are no longer in the repo. After that, sensuctl will run the create command for anything that might have been added to or changed in our repo.

The labels build job works similarly, but it’s going to prune based on a label selector, jenkins_managed == yes. That job is configured to run under the jenkins user. It will run just for resources created by the user called jenkins that has that label using the following prune command:

sensuctl prune checks,handlers,filters,mutators,assets,secrets/v1.Secret --label-selector 'jenkins_managed == yes' -f ${TMPFILE} --config-dir  ${SENSUCTL_CONFIG_DIR}

NOTE: As Jef pointed out, there’s an important point here for good practice. If you’re using sensuctl as part of a CI/CD pipeline, it helps to generate a user for managing resources just for that action. If you do this, then any time you’re using prune to put resources back in line, when you make changes to your Sensu resources inside that repo, the CI/CD action will modify just those resources. Let’s say you’ve changed something or activated a new check in an ad hoc manner, because you were fighting a fire during an incident — so long as you’ve not used the same user as your CI/CD pipeline, these changes won’t get pruned. They can be managed outside your pipeline until such a time as you add them to your repo or delete them all together.

Now let’s move on to the Git repository referenced by these Jenkins jobs. You can see below I have the two different directories I referenced in the two different jobs. One is the labels directory, which has several resource types defined under it and has several checks defined, as does labels.

We can look specifically at some of the resources in the labels directory — for example, check-load — and you can see I have the label, jenkins_managed:"yes" So the prune job in the CI/CD pipeline will know to prune this.

You can see here a check in the username directory of the repo. It doesn’t have the jenkins_managed: "yes" label in its metadata, so those are managed solely off the username.

Once I push this up to GitHub, and the jobs have run, I now have over 750 lines of output from my sensuctl dump command. I prefaced my resource names with labels, so I can easily see them in this sea of output from sensuctl. You can see I have a bunch of checks defined here, and I still have my one cpu check defined because it was defined by a different user, not the pipeline user and it doesn’t have the jenkins_managed: "yes" label on it, so the one check resource I put in manually still exists.

To demo this use case a bit further, if I go back and delete the NTP check and the DNS check, then once I’ve pushed up to GitHub, the DNS check should disappear from usernames, and the NTP check should disappear from labels.

And, if I run my sensuctl dump command again, you can see we’re down to 696 lines of output. So our CI/CD pipeline took care of removing and managing our resources for us.

Watch: sensuctl prune in action

If you’d like to watch these steps as I go through them, check out the demo videos below.

In the first demo, you’ll see me run the prune command against a single CPU check, and then against user-owned resources.

Next, I show how prune works with a file of resource definitions for a couple of Jenkins jobs. It deletes the checks that are no longer in my repo and runs the create command for any resources that have been added to the repo, or changed. I also demonstrate running prune for both the username and labels use cases.

Over to you

sensuctl prune is available as of Sensu Go 5.19 — you can unlock it by downloading the latest version of Sensu. Because prune is an alpha feature, there’s still a lot to be done to help drive its future. We encourage you to give it a go in a test or dev environment, and let us know what you think! Feel free to share your feature requests and feedback in Discourse or create an issue in the Sensu Go GitHub repo.

As always, we invite you to join our Community Forum to stay up to date on the latest Sensu features, webinars, and product releases.

Resources:

• [Blog post] Keeping your configs in good order with sensuctl prune
• Caleb’s webinar from April, in which he demos sensuctl prune and secrets management
• sensuctl prune documentation
• Jef’s sensuctl GitHub action
• Our Discourse post on the webinar, where we welcome you to drop any other questions that may come up