Writing Sensu Plugin Tests with Test-Kitchen and Serverspec

This is a guest post to the Sensu Blog from Ben Abrams, one of the lead maintainers of the Sensu Plugins organization. He offered to share his thoughts in his own words, which you can do too by emailing community@sensu.io.

In my last post I talked a lot about why I love testing infrastructure and the progress we have made in the Sensu community to increase our test coverage. Now I want to walk you through exactly how to setup your environment from scratch to be able to do so.

Start Small

Before we get too far in, I want to remind you to start small. Rather than going OMG test all the things! Too much work! Overwhelmed! Take a step back and decide the most effective type of testing. My simple heuristic is: unit tests for libraries, integration tests for plugins. Once you have decided on the type of testing you should look for what are the most critical things to test. Choose one thing and test it really well. Over time the testing suite will grow and it will become quicker to get through code review as maintainers will be more confident merging something that has a test that passes.

Install dependencies

For plugins we have standardized our integration testing around the following tools:

Test-kitchen: Provides a framework for developing and testing infrastructure code and software on isolated platforms

• https://github.com/test-kitchen/test-kitchen
• https://github.com/test-kitchen/test-kitchen/wiki/Getting-Started
• https://docs.chef.io/kitchen.html
• https://docs.chef.io/config_yml_kitchen.html
• http://kitchen.ci/

Kitchen-docker: Docker driver for test-kitchen to allow using in a lighter weight fashion that traditional virtualization such as vagrant + virtualbox.

• https://github.com/test-kitchen/kitchen-docker

Serverspec: The verification platform. It is an expansion on rspec that allows you to write very simple tests.

• http://serverspec.org/
• http://serverspec.org/resource_types.html
• http://serverspec.org/advanced_tips.html

Setting up your framework

Basically there are a few key components and once you have the required gems installed you will want to start creating a skeleton. You will want to decide the following:

• What platforms do I want to test? For example debian, centos, windows, etc
• What versions of languages do I want to test? For example ruby 2.1, 2.2, 2.3.0, 2.4.0
• What driver you will use for test-kitchen? For example kitchen-docker(preferred), kitchen-vagrant (a bit heavy but super helpful for certain scenarios), kitchen-dokken (a pretty reasonable solutions when using process supervisors), kitchen-ec2 (convenient but costs money), etc

For this example we are gonna test on Debian with several versions of Ruby. This looks something like this in your .kitchen.yml:

https://github.com/sensu-plugins/sensu-plugins-dns/blob/1.3.0/.kitchen.yml#L1-L31

.kitchen.yml configuration

This file has a few essential parts.

Driver

driver:
  name: docker
  use_sudo: false

This is telling kitchen to use the docker driver without the need of sudo to do so. There might be scenarios where you need to use other drivers such as kitchen-dokken with sudo as true when say interacting with systemd.

Provisioner

provisioner:
  name: shell
  data_path: .
  script: test/fixtures/bootstrap.sh

This tells us that we are gonna use a shell provisioner (rather than say chef, puppet, ansible, saltstack, etc) and where to find it.

Verifier

verifier:
  ruby_bindir: /usr/local/bin

This tells kitchen where to find the verifier when using inspec you can omit I believe.

Platforms

platforms:
- name: debian-8

This tells kitchen to use debian 8 as a platform, you could include ubuntu, centos, etc with multiple versions.

Suites

suites:
- name: ruby-21
  driver:
    image: ruby:2.1-slim
- name: ruby-22
  driver:
    image: ruby:2.2-slim
- name: ruby-230
  driver:
    image: ruby:2.3.0-slim
- name: ruby-241
  driver:
    image: ruby:2.4.1-slim

Here we tell kitchen which suites to run which will create a matrix with the platforms desired. This is also telling the driver to use a specific image. In this case we are using docker and want the ruby-slim containers with the appropriate version to test. Therefore the suites are versions of ruby.

Test directory Structure

Under the repository root we expect all the testing artifacts, etc to go in ./test

Here is a snapshot of what the directory structure looks like:

$ tree ./test

./test

├── fixtures

│ └── bootstrap.sh

├── integration

│ ├── helpers

│ │ └── serverspec

│ │ ├── check-dns-shared_spec.rb

│ │ ├── shared_spec.rb

│ │ └── spec_helper.rb

│ ├── ruby-21

│ │ └── serverspec

│ │ └── default_spec.rb

│ ├── ruby-22

│ │ └── serverspec

│ │ └── default_spec.rb

│ ├── ruby-230

│ │ └── serverspec

│ │ └── default_spec.rb

│ └── ruby-241

│ └── serverspec

│ └── default_spec.rb

└── spec_helper.rb

12 directories, 9 files

Under ./test/integration directory we helpers and specific suites. As we want to run all the tests on all suites we write the tests in ./test/helpers/serverspec/ and in the other directories just include the specs. If you wanted to write a specific check for a specific ruby version (say in ruby 2.4.1 they require a new gem to be installed) you can add that to a new spec under the appropriate folder.

Actually writing a test

A really simple example of a test would be does the script execute, return the right exit code, and output matches expected.

An example for DNS:

check = ‘/usr/local/bin/check-dns.rb’

describe command(“#{check} --domain benabrams.it”) do
 its(:exit_status) { should eq 0 }
 its(:stdout) { should match(/DNS OK: Resolved benabrams.it A/) }
end

This tells us that we executed the check, and it resolved the DNS entry and returned that it is healthy. A test look something like this.

A negative test to make sure that things fail when you improperly configure them you might try passing without a domain like this:

describe command(‘/usr/local/bin/check-dns.rb’) do
 its(:exit_status) { should eq 3 }
 its(:stdout) { should match(/DNS UNKNOWN: No domain specified/) }
end

Which would result in a status like this in Travis. Those two things tells us that at least the very basic DNS checking is working.

Running Tests and verification

$ bundle exec kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
ruby-21-debian-8 Docker Shell Busser Ssh <Not Created> <None>
ruby-22-debian-8 Docker Shell Busser Ssh <Not Created> <None>
ruby-230-debian-8 Docker Shell Busser Ssh <Not Created> <None>
ruby-241-debian-8 Docker Shell Busser Ssh <Not Created> <None>

Say you are seeing a failure on Travis on ruby 2.2 but not on others you can run just the 2.2 verification tests like this:

$ bundle exec kitchen test ruby-22-debian-8

You will see a bunch of output which I have removed here but you can see the tests executed:

ruby environment
  behaves like ruby checks
    command “which ruby”
    exit_status
      should eq 0
    stdout
      should match /\/usr\/local\/bin\/ruby/
    command “which gem”
      exit_status
        should eq 0
      stdout
        should match /\/usr\/local\/bin\/gem/

command “which /usr/local/bin/check-dns.rb”
  exit_status
    should eq 0
  stdout
    should match /\/usr\/local\/bin\/check\-dns\.rb/

file “/usr/local/bin/check-dns.rb”
  should be file
  should be executable

command “/usr/local/bin/check-dns.rb --domain benabrams.it”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved benabrams.it A/

command “/usr/local/bin/check-dns.rb --domain sensutest-a.benabrams.it — result 1.1.1.1”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved UNCHECKED sensutest-a.benabrams.it A included 1.1.1.1/

command “/usr/local/bin/check-dns.rb --domain sensutest-a.benabrams.it — result 1.1.2.2”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved UNCHECKED sensutest-a.benabrams.it A included 1.1.2.2/

command “/usr/local/bin/check-dns.rb --domain sensutest-a.benabrams.it — result 1.1.1.1,1.1.2.2”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved UNCHECKED sensutest-a.benabrams.it A included 1.1.1.1,1.1.2.2/

command “/usr/local/bin/check-dns.rb”
  exit_status
    should eq 3
  stdout
    should match /DNS UNKNOWN: No domain specified/

command “/usr/local/bin/check-dns.rb --domain some.non.existent.domain.tld — timeout 1”
  exit_status
    should eq 2
  stdout
    should match /DNS CRITICAL: 0% of tests succeeded: Could not resolve some.non.existent.domain.tld A record/

command “/usr/local/bin/check-dns.rb --domain sensutest-a.benabrams.it — request_count 2”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved sensutest-a.benabrams.it A/

command “/usr/local/bin/check-dns.rb --domain sensutest-a.benabrams.it — request_count 5”
  exit_status
    should eq 0
  stdout
    should match /DNS OK: Resolved sensutest-a.benabrams.it A/

Finished in 2.02 seconds (files took 0.24684 seconds to load)
24 examples, 0 failures

Integrate to CI (Travis)

There are a few conveniences we like to have for our continuous integration.

Rakefile

Add some simple tasks to make life easier:

require ‘kitchen/rake_tasks’

# other tasks that do stuff

# Create the kicthen tasks and alias them for convenience 
Kitchen::RakeTasks.new
desc ‘Alias for kitchen:all’

# when running integration tests you should run all kitchen tests
task integration: ‘kitchen:all’

# when using the quick it skips integration tests
task quick: %i[make_bin_executable yard rubocop check_binstubs]

So a couple things to note here:

• Pull in standard kitchen rake tasks which makes life easy for development
• Define a quick task that has all but integration testing. The reason for this is that we want to by default run all tests but in travis (or any CI for that matter) we use matrix jobs to run tests in parallel speeding up the amount of time it takes for testing to complete in. Therefore we will use quick task plus some extra bash glue to run the right version later in travis.

See this Rakefile for a full example.

.travis.yml

Configuring the configuration file for Travis CI has a lot of important parts to it.

sudo: true
services: docker
language: ruby
cache:
- bundler
before_install:
  - sudo iptables -L DOCKER || ( echo "DOCKER iptables chain missing" ; sudo iptables -N DOCKER )
  - gem install bundler -v 1.15
install:
- bundle install
rvm:
- 2.1
- 2.2
- 2.3.0
- 2.4.1
notifications:
  email:
    recipients:
    - sensu-plugin@sensu-plugins.io
    on_success: change
    on_failure: always
script:
- bundle exec rake quick
- bundle exec rake kitchen:ruby-`echo $TRAVIS_RUBY_VERSION | sed -e "s/\.//g"`-debian-8
- gem build sensu-plugins-dcos.gemspec
- gem install sensu-plugins-dcos-*.gem
deploy:
  provider: rubygems
  api_key:
    secure: PY0CkfyMfmyzQ6J4KQImZ5CTptFUkpYMv2UMullimjHchemJsNtwbFl8jdH+fCKMrMm6RV9keGad7wCQVykM7NaAgPHRbpgiV8ZFYjch7UiBUkxXo2VFaIY53ttlZszJIZZeFRIpmgX3QgDCM4O0Y6apeNHqtQrku6HFKmkBmBmDBKryi5kkgXs7UBOskWk6J5UnM4qzhVy5xnGKs2u/cr/Ls2D1EvI9ADWxpHDygW8vZIwL6daULNTAplRKzJQphDPiuccvQWRxmFE1g7iIhkoxZY3TIumR9dVRXP8xJ6pJ8KUskZIr4iV76gX9tTXEdsofCw7G6HbDwVfdL5FWnsBn9J7v1J7A+IKT0oRqioNMAXRTmjtRb7zgOB6VMDI2hh1x8lMQdPTfiUcoAo8tYEwzNJbn2PU68ufOpAs/gTHMqkixCIaOHwiSubDAa/JmPZmYP8k04sRy1aV3Rfl5IID0d381Ndg8Px/6zEHZMN6MKwmmayDkq14yKodwoPOmWn4D+uNtDgZ1Qcp5WZY8jnPCBF/yeoL+zbNUaF5Oi0Q32jET/kaElhVOAT4mAZPgDFzBAE2OAVlqYZNyaR0NSh0kt9vb1OdlKaZSokXiVrDbEHlwNQ8v4+HDNAS4i4vnakPXhhYGHqKMe/PFE1jEc0AT3h0AYgRV+vcahM0T00E=
  gem: sensu-plugins-dcos
  on:
    tags: true
    all_branches: true
    rvm: 2.1
    rvm: 2.2
    rvm: 2.3.0
    rvm: 2.4.1
    repo: sensu-plugins/sensu-plugins-dcos

Breaking down the important bits:

• We tell travis that this is a ruby project and to test the following versions of ruby: 2.1, 2.2, 2.3.0, 2.4.1
• We do some docker setup
• Run the quick rake tests (all but integration)
• Run the kitchen test for the platforms of choice on the version of ruby that travis is testing
• Build and install gem
• Deploy to rubygems.org with a key, see encrypting variables in travis on tagged commits, testing with the same ruby versions as previously tested in branches and PRs.

For a full example see, review this Travis CI config file.

Expand & Profit

Once you have the most critical tests written and you have the framework to do so in place you can quickly add tests and when you get the appropriate coverage you will notice that the code becomes more stable. I can tell you at my job we have pretty great chef testing using 99% of the same exact tools outlined here. It has given us a great degree of confidence to do gnarly refactors safely, introduce new features without breaking old ones, and that we are not being broken by upstream changes. I have seen at least 4 potential breakages (and one MAJOR outage) in the last month that were prevented by having great testing and integrating it with CI.

You will also know that your PRs will get a lot faster on reviews because as a maintainer I am more likely to scrutinize change when I can’t see that it is actually working with 0 effort.

I hope you will get more involved in the effort! Contribute tests to your favorite plugins and join us on Slack to talk about it with fellow maintainers. We would love to get you involved.