Integrate your IDS + monitoring with the Sensu Tripwire asset

Tripwire, created by our friend Gene Kim, is a popular intrusion detection system (IDS) with both commercial and open source offerings. As a fun side project, I put together a Tripwire asset for Sensu. While this is more a prototype than anything else, I wanted to take this opportunity to offer some background on IDS, Tripwire, and integrating intrusion detection into your monitoring workflow, with the overall aim of illustrating how easy it is to deploy solutions with Sensu.

Intrusion detection systems: a brief overview

Intrusion detection systems — as the name implies — are meant to alert you when something nefarious is happening on your systems, monitoring for malicious activity and policy violations — AKA, a bad actor intruding onto your systems. Unlike an intrusion prevention system (IPS), an IDS doesn’t actively block unwanted traffic or activity, but rather alerts you to their presence, with the idea that you’ll employ your own analysis and take action as needed to protect your network.

An IDS can be network- or host-based (a NIDS or HIDS, respectively) and is a critical part of your broader defense-in-depth strategy. An IDS is not a catchall — there will be false positives (and worse — false negatives!), which can lead to alert fatigue and operator burnout. It’s important to integrate your IDS with a comprehensive monitoring approach so you can fine-tune those alerts (more on that later).

At its most basic level, an IDS works by using file integrity monitoring to identify unwanted or unusual changes to key files that you designate. An IDS will take a snapshot of your file in its desired state, hash that snapshot using an encryption algorithm like MD5, and compare that version to new snapshots, taken at set intervals. If the hashes don’t match, the IDS will trigger an alert.

Intrusion detection with Tripwire

While there are various IDS solutions available, for this post, I’ll focus on one of the more popular solutions: Tripwire. Tripwire is designed to monitor and alert on file and directory changes (AKA, file integrity monitoring, as mentioned earlier). You create a policy file that defines which files and directories you’d like to monitor, as well as associated attributes on those files, such as hashes, file permissions, and ownership. Tripwire will run its checks against the current filesystem based on an established baseline, which you can update at any time to incorporate known, desired changes.

Tripwire has a commercially supported version available at https://www.tripwire.com/ as well as an OSS project: https://github.com/Tripwire/tripwire-open-source. For our purposes, I’ll look at integrating Sensu with the open source Tripwire.

The open source version offers the essential features to get started with intrusion detection. As noted earlier in our general description of IDS, Tripwire will record a system snapshot and, on the next run, compare the current snapshot to the baseline. You can configure OSS Tripwire to generate a single report in which it breaks down the system modifications and the severity of those modifications (which you configure on your end, through your policy file).

For example, perhaps you want to be made aware of any time the /etc/passwd file is changed, so you’d assign it a high severity. While this change could be due to nefarious activity (say, an attacker adding a new account to maintain command and control), it could also be due to your sysadmin adding a legitimate new account. Either way, it’s worth getting alerted on and investigating.

Tripwire reports can be critical to any sort of legal proceeding in which you need to provide irrefutable evidence. By using keys to sign and secure the state file and reports, you can be confident in their integrity — with the great byproduct that it’ll hold up in a court of law.

While OSS Tripwire provides a great framework for IDS, it’s not necessarily scalable across your entire infrastructure (the commercial version takes care of that). But, as I’ll argue below, if you’re already using an observability solution for application and infrastructure health, it makes perfect sense that you’d integrate your IDS alerts alongside your monitoring data.

Monitoring + intrusion detection

I may be biased, but Sensu has some great automation privileges, like assets, which make it easy to install new functionality into your Sensu monitoring environment in a just-in-time fashion. (For more on assets, check out our documentation.) If you’re reading this post, chances are you’re already using Sensu to observe the health of your infrastructure and applications, so why not use it to deploy and manage your intrusion detection system too? In many ways, Sensu is an orchestration platform for command execution — and the reporting of those executions. Essentially, Sensu creates reports through service checks — checking to see if a service is up, down, or flapping, piping the info to standard output, and an exit status to indicate severity. In this way, you’re able to maintain a state engine.

When you apply that concept to intrusion detection, Sensu can take the OSS Tripwire project and wrap it in a check execution plugin, bundling it as an asset. From there, you can create a Sensu check and have it automatically deploy Tripwire to all your machines, build snapshots, and start reporting on file-based modifications.

You can use the Sensu Tripwire asset to monitor the state of your application directory. And, if you’re running immutable infrastructure — building system images and then deploying those images — you can leverage Tripwire to monitor that infrastructure, ensuring it remains immutable (AKA, no unwanted modifications are happening on your systems).

When Tripwire is connected to Sensu, you can also take advantage of the alert component of the Sensu monitoring event pipeline. You can turn the Tripwire report into standard output and a simple exit status code indicating a warning or critical, hook that up to a Sensu handler like Slack or PagerDuty, and leverage that handler for IDS alerting. You can also send time-series data to your TSDB for high-level reporting and send the raw Tripwire reports to a document store like Splunk or Elastic for archival or legal purposes.

Sensu + Tripwire: an example use case

In the following section, I’ll demonstrate an example use case of how you might apply Tripwire + Sensu in your organization. Keep in mind, this is a proof of concept that I threw together in my spare time, and not yet an officially supported project. If this piques your interest, I invite you to contribute to this asset on Bonsai.

There are two main caveats to keep in mind with the example below:

• The policy file I’ve included is hard-coded — it makes assumptions of what’s important to monitor — and as such might not be ideal for your organization.
• The asset runs Tripwire as the sensu-agent user, not as root, which might be problematic for certain files (i.e., there might be constraints of what you can observe as that user, unless you configure a sudo policy to allow it to execute Tripwire). While many folks run Tripwire as root, for this project — scoping Tripwire to just an app directory, and not your whole system — it’s actually better to run Tripwire as a non-privileged user.

Sensu Tripwire is a collection of Sensu assets, packaging up Tripwire OSS (version 2.4.3.7), making it easy to deploy an IDS to systems running the Sensu monitoring agent.

The Tripwire assets are currently compiled for amd64 systems, on Alpine Linux (3.9.5), CentOS (6 and 7), and Debian (Stretch). As noted in the caveats above, the included wrapper shell scripts are opinionated and include Tripwire policy that may be less than ideal for your systems.

Step 1: What’s included

Here’s what the individual components can do.

Initialize a Tripwire database:

tripwire-init.sh

Run a full system check (and initialize the database if it’s missing):

tripwire-check.sh

Run a Tripwire check on a specific web application:

tripwire-check.sh /var/www/*

Step 2: Add the Sensu Tripwire asset to a Sensu Go cluster

sensuctl asset add portertech/sensu-tripwire

Note: The sensuctl asset add functionality was added in Sensu Go 5.13. Download the latest version if you haven’t already, or, if you’re running an earlier version of sensuctl, you can grab the asset on Bonsai.

Step 3: Create a check that uses the Tripwire asset

type: CheckConfig 
api_version: core/v2 
metadata: 
  name: tripwire-web 
spec: 
  command: tripwire-check.sh /var/www/* 
  interval: 30 
  runtime_assets: 
  - portertech/sensu-tripwire 
  subscriptions: 
  - webserver 
  publish: true

Step 4: Combine this project with the Sensu Tripwire Ruby plugin This project can be used in combination with the Sensu Tripwire Ruby plugin project. It’s a plugin used to translate Tripwire reports to Sensu.

sensuctl asset add portertech/sensu-tripwire

sensuctl asset add sensu-plugins/sensu-plugins-tripwire

Here’s an example of how your Sensu check configuration might look:

type: CheckConfig 
api_version: core/v2 
metadata: 
  name: tripwire 
spec: 
  command: check-tripwire.rb --binary tripwire.sh --config-file /tmp/tw/tw.cfg 
  interval: 30 
  runtime_assets: 
  - portertech/sensu-tripwire 
  - sensu-plugins/sensu-plugins-tripwire 
  - sensu/sensu-ruby-runtime 
  subscriptions: 
  - linux 
  publish: true

The above examples — plus additional info, such as available options — can be found on the GitHub repo and the Sensu Tripwire asset on Bonsai. And, if you run into issues using the assets, please open a GitHub issue!

What’s next?

I hope this post gave you an idea of how easy it is to package, share, and try out Sensu solutions with assets. For more on assets (including how to publish an asset with the Sensu Go SDK), check out this post from Developer Advocate Todd Campell, in which he goes over the various community templates we have available that make it even easier to create and share assets. We invite you to try out the Sensu Tripwire asset below — because it exists as an asset in Bonsai (backed by the Bonsai CDN, as well as available on GitHub as a GitHub release), it’s super easy to give it a whirl.

If this proof-of-concept Sensu Tripwire asset piques your interest, please let us know by joining the discussion on Discourse and/or contributing to the existing plugin on Bonsai. We can’t wait to hear what you come up with!