Organize your monitoring fleet with Sensu cluster federation

Recently, I led a webinar on Sensu cluster federation and some of the ways users can effectively use Sensu’s API.

With the API, you can create as many clusters as needed and federate them without much effort. Also, Sensu makes the management of these clusters very easy by allowing you to manage access using a single web UI.

In this post, I will recap the webinar, with step-by-step demos that will touch on how you can:

• Set up multiple clusters
• Tie clusters together and manage them in one place.
• Implement role-based access control (RBAC) to protect sensitive data
• Enable Transport Layer Security (TLS) to encrypt communication between clusters

Sensu federation overview

The core idea in any federated software system is that you have similar systems (usually geographically separated) that need to work both independently and in a coordinated fashion. Federated systems are not exact replicas of each other; they have unique local states that can’t easily be merged, but you still want to be able to have visibility into their operation from a central location. Federation encompasses how these mostly independent systems interconnect and can be coordinated from a central location.

In the case of Sensu, federation makes it possible for different Sensu backend clusters to share some resource configurations while processing events independently. From an operator perspective, this makes it possible to have visibility into all events in all clusters in a single web UI and coordinate resource changes for any cluster in the federation.

Sensu federation works by making use of signed authentication tokens together with a role-based access control policy to allow authorized users logged into the federation gateway UI to interact with each of the clusters in the federation.

The Sensu Federation API makes it easy to add, edit, and remove clusters from a federated environment using sensuctl (our command-line tool) on your gateway backend cluster. Once added to the federation, instead of managing each cluster separately, you can use the gateway cluster’s web UI to manage all federated clusters (if the RBAC policy for your user allows it!). Any changes you make from the gateway cluster will go into effect on the other federated clusters.

How federation in Sensu Go works

Federation in Sensu Go is very easy to navigate. You can create clusters, web tokens, and manage multiple datacenters with minimal effort. Shortly we’ll walk through the step-by-step process on creating a highly secured federated network using Sensu Go.

Let’s dive in, shall we?

Preparing your clusters for assimilation

First of all, you need to prepare your clusters and get them ready to integrate and communicate with each other securely across the network. To ensure this, you need to:

• Configure clusters for secure etcd communications
• Prepare shared token signing keys
• Set up RBAC resource etcd replicators
• Register downstream clusters with the gateway cluster

Demos: getting your cluster ready for assimilation on the web UI

The demos assume you have all the following prerequisites:

• Two publicly addressable Sensu clusters that are up and running with embedded etcd.
• TLS certs pre-configured and in use for API, web, & agent sockets (Let’s Encrypt is a great service to use to generate trusted certs if you own a domain).
• Firewall/security-group rules that will facilitate secured communication between clusters.

This demo will walk you through how you can get your clusters ready for assimilation on the Sensu web UI.

Starting Gateway configuration

The following demo touches on how you can configure your gateway server. To prep for the demo, I’ve already configured the firewall, security groups, and Let’s Encrypt certs, so all I’m doing is getting the backend up and running:

Starting first remote cluster configuration

I’ll do the exact same configuration for a remote cluster. Once configured to be part of the federation, my goal is to not have to interact directly with the remote cluster, but to service it via the federated view in the gateway cluster’s web UI. The only thing I do differently is I’m starting with a different admin user:

Note: On my gateway server backend, I have an admin user called webinar while on the remote backend server, the admin user is called admin.

Preparing embedded etcd for cross-cluster communication

To ensure secure cross-cluster communications, you must ensure that all the etcd clusters are configured correctly. Additionally, if you are using the embedded etcd you need to generate your own non-default values for the etcd-advertise-client-urls and etcd-listen-urls especially when you’re on a network you don’t control. The default values for the embedded etcd are restricted to localhost access, so they won’t work in a federated environment. If you are already using an external etcd for your Sensu clusters you have most likely already addressed this issue.

When you’re configuring your embedded etcd, make sure you take note of the following:

TLS configuration options:

• etcd-cert-file
• etcd-key-file
• etcd-trusted-ca-file
• etcd-client-cert-auth (use to enable mutual TLS)

etcd replication options:

• etcd-advertise-client-urls
• etcd-listen-client-urls

Note: You always have the option to use an external etcd cluster instead of the embedded etcd that Sensu backend gives you out of the box. If you are using an external etcd, then you should be able to transition these instructions accordingly.

Demos: preparing embedded etcd

To prepare embedded etcd for cross-cluster communication, you’ll need to create etcd TLS certificates, configure embedded etcd for each Sensu backend to use those certificates, and prepare the shared signing token keys. We illustrate these steps in the following demos.

Create etcd TLS certificates using a private CA

In the demo below, I create the necessary Certificate Authority (CA) along with the certs the etcd cluster will need using the cfssl tool. When creating the certs, I make sure to populate the SAN extension for each cert with the publicly addressable hostname for the server. By doing so, the entities can verify each other without any difficulty:

See our documentation for the steps to generate private certs.

Configure embedded etcd for each Sensu backend to use TLS certificates

Once I’ve created the necessary etcd TLS certificates, I update the backend config and then restart the backend to effect the changes:

Preparing the shared token signing keys

The next thing we need to do is to have consistent shared signing keys for both clusters. Each backend must have the same private key (jwt-public-key-file) and public key (jwt-private-key-file) that will ensure token-based user access across all clusters. Once you log in to the gateway, an authentication token will be issued using the shared signing key. The web UI will use that token to interact with all federation clusters. The shared signing key makes the token authentication work seamlessly for all federation members.

Demos: generating public/private signing keys + configuring backends to use JWT signing key

Generate public/private signing key for JWTs

In this demo, I use open SSL and sensuctl to generate a private/public key pair:

Configure all backends to use common JWT signing key

Once I’ve generated the key pair, I configure it on both the alpha and the gateway clusters so that both of them can use the same JWT for authentication. When this is done, I restart the backend to effect changes:

Replicate Sensu RBAC to downstream clusters

Now that we have the etcd set up and the shared JWT is properly configured, the next step is to focus on role-based access controls (RBAC) on the gateway and all the other federated servers.

You’ll need an RBAC policy that will allow authenticated users to take actions on the gateway clusters and effect it on the other federated clusters.

However, instead of having to create this policy manually, the Sensu Federation API allows you to create resource replicators. With these replicators, etcd will mirror your changes into all the downstream policies you want to replicate.

Demo: Creating a read-only user and related cluster role-bindings

In this demo, I create a user just for federation viewing only. This user will not be able to make any operational changes.

I create the user along with the role binding that states that they can only use the view role at the cluster level. Also, I create the replicator to replicate this command from the gateway cluster to the alpha cluster.

Register downstream clusters

Registering your downstream Sensu clusters will establish that the additional clusters exist on the web UI. You will be able to login to your gateway cluster and use JWTs to authenticate to each cluster that’s registered on the downstream.

In the demos below, I register the alpha cluster with the gateway, so the federated web UI will be able to communicate with it using JWTs:

Now, for the fun part. With all the pieces in place, I log into the gateway web UI as the federation viewer and see both the alpha and the gateway clusters (the blue and orange boxes in the top right):

The user is only defined in the gateway; I can’t login directly to the alpha web UI using the federation-viewer login:

Using a cluster admin user defined only on the gateway gives you the ability to create resources in either the alpha or gateway clusters. In the following demo, I create a simple echo test check; after it executes, you can see it under the events in the alpha cluster but it doesn’t show up in the gateway cluster because that check doesn’t exist in the gateway cluster. These namespaces are still separated. Even though they’re sharing role-based access controls for the ClusterRoles & ClusterRoleBindings, they aren’t sharing anything else.

Taking the power of resource replication further

Each Sensu resource can be replicated. However, it only makes sense to replicate certain resources like checks and handlers. Also, you can replicate resources from a specific namespace defined in the gateway to a downstream cluster:

Please note: Replication is one-way only! Local changes made in the downstream cluster won’t propagate back to the gateway.

Go boldly and learn more

I hope this post gave you a sense of how to create a fully federated environment using Sensu cluster federation. For further learning, check out these resources:

Scaling Sensu clusters with etcd

• Webinar: Scaling your monitoring with Sensu clustering
• Blog post: How to scale your monitoring with Sensu clustering

Sensu RBAC

• Webinar: Introduction to RBAC in Sensu Go
• Blog post: Intro to RBAC in Senso Go

Creating certificates and securing Sensu

• Guide: Generate certificates for your Sensu installation
• Guide: Secure Sensu

The Sensu federation API

• The Sensu federation API documentation