DevOps tools for compliance monitoring

Monitoring and compliance are, in many ways, synonymous. At the very least, there’s a big overlap in terms of defining and monitoring rulesets you care about. The time frame may vary; with monitoring, you might jump on an alert right away, as opposed to the compliance team’s quarterly audit, but the foundation remains the same. As our development cycles grow ever more dynamic, the need for automating repetitive tasks becomes all the more important.

In this post, I’ll examine how you can align your compliance and monitoring efforts — automating manual processes where possible — drawing from Sensu and InSpec as examples. The ultimate goal is to get your teams on the same page — security and ops working together to ship high-quality, secure applications.

A DevOps pipeline for compliance

Compliance, at its most basic, is the action or fact of making sure you’re in line with previously agreed-upon rules and standards (for example, the guidelines set by the Payment Card Industry Security Standards Council in order to maintain PCI compliance). Once those standards are agreed upon, your compliance efforts including tracking and reporting on whether you’re meeting — or not meeting — those standards.

DevOps practices increase the velocity of software delivery, automating processes that were previously manual (and therefore time-consuming). The idea behind applying a DevOps mindset to compliance work is to automate what you can, saving time, reducing errors, and achieving consistency in your operations. By automating compliance and governance, organizations can actually increase the integrity of their assets and improve their overall security for their entire organization.

CapitalOne, for example, adopted a DevOps pipeline — outlined in detail in this post — that allowed them to deliver high-quality software faster, resulting in fewer security flaws. They designed their pipeline such that it continues to produce a wealth of data points, so they can continuously audit and improve it.

Traditionally, ops and security folks were not on the same team (figuratively and literally), which made collaboration difficult. Now, with the advent of movements like DevSecOps, operators and security folks can support each other in the development and deployment of applications, empowering the automation of manual tasks for better performance and security. Now, they have a shared toolbox of code from which the compliance team can determine whether the company is in compliance, and from which the ops folks can produce automated processes.

Before we dive into our examples of DevOps for compliance in practice, a caveat: every organization is different, and as such whether it makes sense to automate a task (and the approach you take to monitoring and alerting) hinges on what part of the business is driving the decision you make around what needs human attention. Take your corporate website as one use case: if your website’s being down means you lose hundreds of thousands of dollars a minute (and therefore, your team needs to be super responsive to that event), that should inform how you approach monitoring and what kind of alerts reach people. On the other side of the coin, maybe you’re in an organization where the website’s being down is not a critical incident in terms of revenue impact, but every quarterly audit you fail has a higher price tag associated with it. In turn, that should inform how you approach your use of DevOps tools for compliance.

Tools like InSpec and Sensu — which I’ll dive into below — are flexible enough to meet your needs, whatever they may be. Ultimately, it’s all about making code reuse possible and get your ops and compliance teams on the same page.

DevOps tools for compliance monitoring

Tools like InSpec allow you to write monitoring checks using domain-specific language. For monitoring, that might look like writing a check that makes an HTTP request to a web server — if it doesn’t return a 200/OK response, raise an alert. For checking compliance, that might look like compliance controls (implemented through a compliance profile), which are designed to support existing policies and procedures aimed to reduce risk.

Below, I’ll offer a brief overview of both InSpec and Sensu, as well as an example use case of both tools working together for compliance monitoring.

InSpec overview

Chef InSpec is an open source automated testing tool for integration, compliance, security, and other policy requirements.

InSpec comes in really handy for testing whether or not your systems and processes are in compliance in an automated fashion. You can write up a body of assertions, saying, for example, “This service is running and listening on this port” or “This file should be readable only by the root user.” You run InSpec at one point in time and it tests those assertions, telling you where you’re in compliance and where you’ve failed. Again, it’s all about automating manual processes. From the InSpec perspective, automation means your systems are checking everything you’ve defined as a control when writingInSpec profiles.

Sensu overview

If you’re reading this blog post, chances are you’re already familiar with Sensu. Sensu Go, the latest version of Sensu, is designed to be more portable, easier and faster to deploy, and (even more) friendly to containerized environments. Check out our documentation for a complete overview, as well as our getting started guide.

Putting it together: compliance monitoring with the Sensu InSpec plugin

Ops and security joining forces, as noted earlier, empowers both groups to accomplish their goals more quickly and safely. By using a monitoring tool like Sensu to run your InSpec reports at set intervals, you can generate alerts on what you care about and prioritize issues that need to be seen and evaluated by humans. And while not every compliance report is automatable or appropriate for monitoring, there’s a huge overlap between compliance and monitoring — between the parts of the business that are trying to satisfy compliance concerns and those that are concerned with performance. Just like the shared toolbox I mention earlier, there’s information and processes you can reuse in other areas of the business. To flip that on its head, another use case is the monitoring team has come up with information and can use Sensu to help the compliance team add a few checks to keep an eye on compliance details.

Folks in our community are already doing this in various ways; see this talk from Paul Czarkowski at Sensu Summit 2017, in which he describes how his company at the time, IBM, started writing InSpec tests before they “realized they could have Sensu use InSpec tests as checks.”

A Sensu plugin — in general — encapsulates knowledge of how to take a measurement or action, like making an HTTP request, and the logic to determine whether the result of the action is within expected parameters (i.e., if HTTP response isn’t 200 OK, exit with an error code of 2). InSpec’s domain language can be used to implement many of the measurements or actions which have found their way into various check plugins over the years, but in many cases you’re dependent on what the language allows. But, since Sensu allows you to write plugins in whatever language you choose, a little bit of wrapper code around InSpec makes it easy to reuse InSpec profiles in the context of monitoring.

I created this proof-of-concept plugin that — as a check — executes InSpec profiles and generates a Sensu event for each non-passing control. Sensu takes those InSpec assertions, runs them, and every failure will generate a Sensu event. The plugin is available as an asset in Bonsai, the Sensu asset index, ready for you to download and try.

Dynamic runtime assets are the best way to use the InSpec plugin. If you’re using Sensu Go 5.13 or later, you should be able to use the following command to add the asset:

sensuctl asset add sensu-plugins/sensu-plugins-inspec

And here’s what a check definition might look like:

type: CheckConfig
spec:
  command: "check-inspec.rb -d /tmp/my_inspec_profile -l example_handler"
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  publish: true
  runtime_assets:
  - sensu-plugins/sensu-plugins-disk-checks
  - sensu/sensu-ruby-runtime
  subscriptions:
  - linux

Head to Bonsai for the full guide, and let us know what you think, either by creating an issue in GitHub or posting in our Community Forum.

Over to you

We hope this post gave you an idea of how you might apply a DevOps mindset to your compliance monitoring and — at the same time — gave you a sense of how easy it is to spin up new solutions with Sensu. Give the Sensu InSpec plugin a try — we can’t wait to hear what you think!