Securing Your Monitoring Software With mTLS

Mutual transport layer security (mTLS) is an important subject among security, reliability, and engineering professionals who need to secure API communication as well as communication between machines and the applications and services they run. And for good reason: in 2022, the global average cost of a data breach was US$4.35 million^[1], and almost double that in the United States at US$9.44 million. In addition to preventing exposure to the expense of a data breach, communication security is vital to preventing audit penalties.

mTLS authentication offers a security solution for everything from network connections and data transfers to controlling service traffic with a service mesh. This article explains what mTLS is, reasons to use mTLS, how mTLS works in general, and how to configure mTLS for your Sensu instance.

What Is mTLS?

Briefly, mTLS ensures that both sides of a connection, client and server, are secure and trustworthy (i.e., they are who they say they are). mTLS achieves this by validating each side’s private keys and the information in their TLS certificates.

mTLS helps companies avoid unnecessary security risks. It’s often part of a Zero Trust security model for validating communication via enterprise applications, connections, and servers.

mTLS vs. TLS

mTLS builds on the transport layer security (TLS) protocol. You’re using TLS every time you connect to a secure website via your web browser. In this scenario, TLS guarantees that your browser can trust the visited site’s identity, the data exchanged during your visit is confidential, and the data you receive is identical to the data the site sent. TLS does this by use of a certificate authority (CA), which issues a digital certificate to the site’s owner based on proof that they own the domain. Your browser knows it can trust the site because it trusts the CA.

In this scenario, TLS is sufficient for securing communication because the communication is one-way: from the web server to you. Your browser only needs to authenticate the server’s identity. Securing both sides of the communication so that the server can also authenticate the client’s identity requires mutual TLS, or mTLS.

So mTLS is essentially TLS on both sides of the communication: server and client. mTLS requires both sides to use certificates to authenticate, an additional measure that is necessary for securing many types of network communication.

Reasons to Use mTLS

For any software, it’s crucial to minimize all exposed attack surfaces. mTLS authentication guarantees the communication from both server and client is authentic, confidential, and complete. Without these guarantees, the communication is vulnerable to falsification, inappropriate disclosure, and modification. The requirement for authentication from both server and client means mTLS prevents a variety of attacks, including:

• Spoofing, where attackers imitate a user or web server.
• On-path attacks, where attackers establish themselves between the client and server so they can intercept or modify communications.
• Credential stuffing, where attackers use credentials obtained in a data breach to log in as an actual user.
• Brute force and phishing attacks, where attackers guess or steal a user’s credentials and use them to log in as or otherwise impersonate the user.
• API attacks, where attackers send malicious API requests intended to exploit a vulnerability or change the way the API works.

With mTLS, in all of these cases, the attacker would still need a TLS certificate and a corresponding private key to authenticate from the client side. User credentials alone are not enough to authenticate and gain access if mTLS is configured.

In addition to preventing attacks, mTLS is useful any time you need to confirm a specific user’s or device’s authenticity and validity. Examples include:

• Devices that connect to your company’s network, as well as users who log in to your network and applications that run on it.
• Data exchanges via API.
• Microservices that require authentication for all cross-service communication.
• Smart cards for physical access to company buildings.
• Internet of Things (IoT) and unmonitored devices like security cameras.

Within Sensu, you must secure Sensu agents with mTLS to use secrets management.

In short, mTLS is useful when you need to confirm client identity and secure two-way communication. But there’s another consideration in implementing mTLS: understanding the keys and certificates it requires.

How does mTLS work?

mTLS follows this basic sequence of events:

1. The client connects to the server
2. The server provides its TLS certificate to the client
3. The client authenticates the server’s certificate
4. The client provides its TLS certificate to the server
5. The server authenticates the client’s certificate
6. The server grants access to the client
7. The client and server exchange information over the encrypted TLS connection

To establish the encrypted connection under mTLS protocol, both client and server must have a public/private key pair and a TLS certificate. The TLS certificate is a digital X.509 certificate, and the public key is included with it. The public key identifies the certificate’s issuer and expiration date. The public key has a corresponding private key, which is not included in the TLS certificate.

In a process known as the TLS handshake, the server provides its TLS certificate to the client and uses the private key to confirm the server’s identity to the client. The same process happens on the other side, with the client providing its TLS certificate for verification by the server.

Configure Sensu agent mTLS authentication

When you deploy Sensu for use outside of a local development environment, you should configure mTLS to secure communication between Sensu backends and agents as well as communication between human operators and the Sensu web UI and sensuctl. This section explains how to use mTLS to secure Sensu communications for a clustered backend and agents.

To follow the instructions in this section, you must have Sensu installed on a single backend or three backends that you plan to cluster together, as well as one or more agents. You will need to install TLS and generate a CA as well as self-signed certificates from that CA for the backend and agent. The examples assume that you’ve installed the certificates and keys in the /etc/sensu/tls directory on a Linux system.

After creating a CA, use it to generate certificates and keys for each backend server. For each backend server, you’ll need the IP addresses and hostnames to use in backend and agent communications. Then, use your CA to generate a certificate that agents can use to connect to the Sensu backend. For the agent certificate, you only need to specify a CN, not an address.

WARNING: Before you use the certificates to secure Sensu with mTLS, delete the ca-key.pem file from the directory that contains your CA and certificates. This file contains sensitive information and is not needed to continue.

For example, if you have three backend servers, the directory listing would include 15 files (after deleting ca-key.pem):

/etc/sensu/tls/
├── agent-key.pem
├── agent.pem
├── agent.csr
├── backend-1-key.example.com.pem
├── backend-1.example.com.pem
├── backend-1.example.com.csr
├── backend-2-key.example.com.pem
├── backend-2.example.com.pem
├── backend-2.example.com.csr
├── backend-3-key.example.com.pem
├── backend-3.example.com.pem
├── backend-3.example.com.csr
├── ca.pem
├── ca-config.json
└── ca.csr

Finally, install the CA certificates. With the certificates installed, you can secure your Sensu installation with mTLS to make your Sensu installation production-ready.

Secure etcd peer communication

NOTE: This section applies only for Sensu 6.x and earlier.

WARNING: Update the default configuration for Sensu’s embedded etcd with an explicit, non-default configuration to secure etcd communication in transit. If you do not properly configure secure etcd communication, your Sensu configuration will be vulnerable to unauthorized manipulation via etcd client connections.

To properly secure etcd communication, replace the default configuration option values in your backend store configuration in /etc/sensu/backend.yml as follows:

1. Replace the placeholder with the path to your certificate and key for the etcd-cert-file and etcd-key-file to secure client communication:

etcd-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
etcd-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

2. Replace the placeholder with the path to your certificate and key for the etcd-peer-cert-file and etcd-peer-key-file to secure cluster communication:

etcd-peer-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
etcd-peer-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

3. Replace the placeholder with the path to your ca.pem certificate for the etcd-trusted-ca-file and etcd-peer-trusted-ca-file to secure communication with the etcd client server and between etcd cluster members:

etcd-trusted-ca-file: "/etc/sensu/tls/ca.pem"
etcd-peer-trusted-ca-file: "/etc/sensu/tls/ca.pem"

4. Add non-default values for etcd-listen-client-urls, etcd-listen-peer-urls, and etcd-initial-advertise-peer-urls:

etcd-listen-client-urls: "https://localhost:2379"
etcd-listen-peer-urls: "https://localhost:2380"
etcd-advertise-client-urls: "https://localhost:2379"
etcd-initial-advertise-peer-urls: "https://localhost:2380"

NOTE: If you are securing a cluster, use your backend node IP address instead of localhost in the non-default values for etcd-listen-client-urls, etcd-listen-peer-urls, and etcd-initial-advertise-peer-urls.

5. Set etcd-client-cert-auth and etcd-peer-client-cert-auth to true to ensure that etcd only allows connections from clients and peers that present a valid, trusted certificate:

etcd-client-cert-auth: "true"
etcd-peer-client-cert-auth: "true"

Secure the Sensu agent API, HTTP API, and web UI

The Sensu Go agent API, HTTP API, and web UI use a common stanza in /etc/sensu/backend.yml to provide the certificate, key, and CA file needed to provide secure communication.

NOTE: By changing these configuration options, the server will communicate using transport layer security (TLS) and expect agents that connect to it to use the WebSocket secure protocol. For communication to continue, you must complete the configuration in this section and in the Sensu agent-to-server communication section.

Configure the following backend secure sockets layer (SSL) attributes in /etc/sensu/backend.yml:

1. Replace the placeholders with the paths to your CA root, backend certificate, and backend key files for the trusted-ca-file, cert-file, and key-file configuration options:

trusted-ca-file: "/etc/sensu/tls/ca.pem"
cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"

2. Set the insecure-skip-tls-verify configuration option to false:

insecure-skip-tls-verify: false

3. When you provide these configuration options, the agent WebSocket API and HTTP API will serve requests over SSL/TLS (https). For this reason, you must also specify https:// schema for the api-url configuration option for backend API configuration:

api-url: "https://localhost:8080"

4. Restart the sensu-backend service:

sudo systemctl restart sensu-backend

After you restart the sensu-backend service, the configuration options will load and you will able to access the web UI at https://localhost:3000. Configuring these options will also ensure that agents can communicate securely.

You can use the same certificates and keys to secure etcd, the HTTP API, and the web UI. However, if you prefer, you can use a separate certificate and key for the web UI (for example, a commercially purchased certificate and key).

Secure Sensu agent-to-server communication

By default, an agent uses the insecure ws:// transport. Here’s an example for agent configuration in /etc/sensu/agent.yml:

backend-url:
  - "ws://127.0.0.1:8081"

To use WebSocket over SSL/TLS (wss), change the backend-url value to the wss:// schema in /etc/sensu/agent.yml:

backend-url:
  - "wss://127.0.0.1:8081"

NOTE: If you change the agent configuration to communicate via WebSocket Secure protocol, the agent will no longer communicate over a plaintext connection. For communication to continue, you must complete the steps in this section and secure the Sensu agent API, HTTP API, and web UI.

You can also provide a trusted CA root certificate file as part of the agent configuration (named ca.pem in the example for creating a CA). If you will start the agent via sensu-agent start, pass the --trusted-ca-file flag with the start command. Otherwise, include the trusted-ca-file configuration option in the agent configuration in /etc/sensu/agent.yml:

trusted-ca-file: "/etc/sensu/tls/ca.pem"

Configure Sensu agent mTLS authentication

By default, Sensu agents require username and password authentication to communicate with Sensu backends. When agent mTLS authentication is enabled, agents do not need to send password credentials to backends when they connect.

When using mTLS authentication, agents do not require an explicit user in Sensu. Sensu agents default to authenticating as the agent user and using permissions granted to the system:agents group by the system:agents cluster role and cluster role binding.

NOTE: Sensu agents need to be able to create events in the agent’s namespace. To ensure that agents with incorrect CN fields can’t access the backend, remove the default system:agents group.

For example, if you have a certificate named client.pem, run the following command to view the certificate’s CN with openssl:

openssl x509 -in client.pem -text -noout

The response should be similar to this example:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:

37:57:7b:04:1d:67:63:7b:ff:ae:39:19:5b:55:57:80:41:3c:ec:ff
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = CA

        Validity
            Not Before: Sep 26 18:58:00 2019 GMT
            Not After : Sep 24 18:58:00 2024 GMT
        Subject: CN = client
...

The Subject: field indicates the certificate’s CN is client, so to bind the agent to a particular user in Sensu, create a user called client.

To enable agent mTLS authentication:

1. Add the following configuration options and values to the backend configuration /etc/sensu/backend.yml:

agent-auth-cert-file: "/etc/sensu/tls/backend-1.example.com.pem"
agent-auth-key-file: "/etc/sensu/tls/backend-1-key.example.com.pem"
agent-auth-trusted-ca-file: "/etc/sensu/tls/ca.pem"

2. Add the following configuration options and values to the agent configuration in /etc/sensu/agent.yml:

cert-file: "/etc/sensu/tls/agent.pem"
key-file: "/etc/sensu/tls/agent-key.pem"
trusted-ca-file: "/etc/sensu/tls/ca.pem"

Nicely done! Your Sensu installation should now be secured with mTLS. For even more detailed instructions, read the Generate certificates and Secure Sensu pages in our documentation. When you’re ready to finish deploying Sensu, read Run a Sensu cluster. Clustering improves Sensu’s availability, reliability, and durability. With a healthy clustered backend, you can absorb the loss of a backend node, prevent data loss, and distribute the network load of agents, as well as replicate changes to all cluster members.