Monitoring Kubernetes, part 4: the Sensu native approach

This is part 4 in a series on monitoring Kubernetes and Docker. Catch up on parts 1, 2, and 3.

At this point in our series, you’re likely quite familiar with the many opportunities and challenges that Kubernetes presents (especially when it comes to monitoring!). The last couple of posts take at a look at Prometheus for monitoring Kubernetes, with a side-by-side comparison with Sensu, and illustrate how they work in tandem. In this post, I’ll show you the Sensu-native approach to monitoring Kubernetes, as well as walk you through monitoring your applications running on Kubernetes, using examples from the sensu-kube-demo repository.

Monitoring Kubernetes with Sensu

While Sensu + Prometheus is a great option (more on that in part 3), you can also use Sensu on its own to monitor Kubernetes. Several folks in our community are already doing so — whether that’s an OG approach with Sensu 1.x (see this post from Community Member Andy Repton and watch Lee Briggs’ Sensu Summit 2018 talk) or with Sensu Go (which, as it just so happens, is designed with Kubernetes in mind — e.g., our RBAC and sensuctl, the command-line tool). The patterns I’ll delve into apply to both Sensu 1.x and Go (and you’ll see our demo repo has example configs for both) but we’ll focus on Go for our examples, since Sensu 1.x is reaching EOL at the end of this year.

Of the data sources I mention in part 1, kube-state-metrics gives you most of what you need in terms of the Kubernetes cluster state, without inundating you with too much information. (It’s a polarizing topic, but there is such a thing as too much data; my experience has taught me that a curated approach to monitoring makes better use of resources and combats alert fatigue.) Kube-state-metrics collects information on the current state of your containers. There are three ways to plug this information into your monitoring event pipeline:

• Use the sidecar pattern to deploy a Sensu agent alongside your application container. Kubernetes pods can contain the container (say that five times fast!) running your application along with other containers running supportive applications such as the Sensu agent. These all share the same network space so your applications can talk to Sensu as if they were running in the same container or host. The agent can simultaneously talk to localhost in the same fashion — for example, if it’s in the pod with a web app, it can check localhost to make sure it’s up and running and healthy.

• Run a daemonset for the Sensu agent that runs the Sensu agent on Kubernetes. This daemonset will deploy a Sensu agent per Kube node that can also do these responsibilities.
• Run a Sensu agent on the Kube host (the actual physical host or virtual machine) that runs all the containers. It can inspect the Kubernetes services to make sure they’re running as well as collect data from kube-state-metrics, local services exporting metrics, and a local node exporter (if you’re running one). In this use case, you’d be running the Sensu agent outside of Kubernetes.

The most flexible way for monitoring Kubernetes is the sidecar pattern. My advice is: use the sidecar pattern for your application monitoring and use a Sensu agent on all of the Kube hosts for monitoring Kubernetes itself.

If you have full control of the hardware, run Sensu outside of Kubernetes so you can inspect the state of Kubernetes safely. You shouldn’t monitor Kubernetes from Kubernetes, because if Kubernetes goes down, so does your monitoring.

For an example of monitoring Kubernetes with Sensu, check out our sample app, which walks you through setting up multitenancy, creating a Sensu pipeline to Slack and InfluxDB, and collecting Kubernetes metrics.

How to monitor your applications running on Kubernetes

In addition to monitoring Kubernetes itself, you’ll want to keep tabs on your applications running on Kubernetes. The aforementioned daemonset and sidecar method comes into play: in the preceding section I outline how you can run a Sensu agent daemonset which would set an agent on every Kubernetes host. You can make that agent’s API available to local containers, so that your applications can emit data to that daemonset agent. Another approach is to have that daemonset agent discover and monitor services on that particular host — for example, you discover several web servers running on your localhost, and check HTTP against them. If they go away, but the container is still scheduled, then there’s a failure.

If you go this route, we want to hear about it! Join us in Slack and Discourse to share what you’ve learned with the community.

Going forward: a multi-generational future

I hope you found this series useful in terms of starting to reason about monitoring Kubernetes and containerized infrastructure. As infrastructure grows increasingly complex — spanning multiple generations of technology — it’s more critical than ever that we keep pace in terms of our tooling and practices. One of our main goals at Sensu is to truly future proof your monitoring, preparing you for what’s next while bridging the gap from legacy infrastructure, to give you total visibility — from Kubernetes to bare metal.

Join Sensu CEO Caleb Hailey for a webinar on July 10 @ 10am PT, where he’ll demo monitoring Kubernetes workloads and clusters.