Prometheus Monitoring using the Sensu Prometheus Collector

I am very excited to share with you a little project that I started on Monday, the Sensu Prometheus Collector for monitoring prometheus endpoints! I have wanted to create something like this for a long time now and I finally found the time to do it.

The Sensu Prometheus Collector is a Sensu Check Plugin that collects metrics from a Prometheus exporter, metric HTTP endpoint, or the HTTP Query API. The collected metrics are outputted to STDOUT in one of three formats: Influx (the default), Graphite, or JSON. Sensu can then deliver the outputted/collected metrics to one or more time-series databases, like InfluxDB or Graphite! Choose your TSDB!

How Sensu Complements Prometheus

The Sensu Prometheus Collector turns Sensu into a SUPER POWERED Prometheus metric poller, leveraging Sensu’s PubSub design and client auto-registration. Sensu is able to automatically discover and begin polling metric HTTP endpoints, securely transporting collected metrics to a desired time-series database. Sensu’s client connections traverse complex network topologies, including those that use NAT and VPNs, eliminating any need to punch holes in firewalls for metric scraping.

Usage Examples

Operating the Sensu Prometheus Collector is very straightforward and configuring a Sensu Check to run it is easy. Simply poll metrics from a Prometheus exporter or an application’s metric HTTP endpoint and output them in the Influx Line Protocol format:

$ sensu-prometheus-collector -exporter-url \ http://localhost:8080/metrics
go_memstats_last_gc_time_seconds value=0 1506991233
go_memstats_mspan_sys_bytes value=32768 1506991233
...

Sensu Configuration Examples

A Sensu Check named “prometheus_metrics”, that collects metrics from a metric HTTP endpoint on application tier systems, and sends them to a Sensu Event Handler named “influx”:

{
 "checks": {
   "prometheus_metrics": {
     "type": "metric",
      "command": "sensu-prometheus-collector -exporter-url http://localhost:8080/metrics",
      "subscribers": ["app_tier"],
      "interval": 10,
      "handler": "influx"
    }
  }
}

A Sensu Event Handler named “influx”, that sends collected metrics in the Influx Line Protocol format to a Influx UDP listener:

{
   "handlers": {
     "influx": {
       "type": "udp",
       "mutator": "only_check_output",
       "socket": {
         "host": "influx.example.com",
         "port": 8189
      }
    }
  }
}

Hopefully these examples helped you get an idea of how the Sensu Prometheus Collector works and how to configure Sensu to use it.

The full power of the Sensu Prometheus Collector can be found in the help output:

$ sensu-prometheus-collector -h
Usage of ./sensu-prometheus-collector:
 -exporter-url string
 Prometheus exporter URL to pull metrics from.
 -metric-prefix string
 Metric name prefix, only supported by line protocol output formats.
 -output-format string
 The check output format to use for metrics {influx|graphite|json}. (default “influx”)
 -prom-query string
 Prometheus API query string. (default “up”)
 -prom-url string
 Prometheus API URL. (default “http://localhost:9090”)

For more examples and documentation, please check out the project’s README.

Contributing

The Sensu Prometheus Collector is very new (and raw), making for many opportunities to contribute to the project! I have begun to create GitHub issues for desirable features and enhancements, labeling some of them with “good first issue” to help identify issues that are more approachable. I hope you find this little project as exciting as I do, help me improve it and give it the polish that it deserves!

Try It!

The Sensu Prometheus Collector is rad and ready for you to try! Use Prometheus instrumentation libraries and exporters with Sensu and store the collected metrics in your favourite time-series database. The more of us using the plugin and experimenting, the better!

The Sensu Prometheus Collector for Prometheus Monitoring can be found on my GitHub.