TLS monitoring

The following is a guest post from Sensu Community Maintainer Eric Heydrick.

Uh oh, the site’s certificate has expired. How do we generate a new one? Where’s the private key? Which servers need the new cert? What even goes in the cert? If this sounds all too familiar, rest assured you’re not alone. Outages due to expired certs are far too common and it happens to sites of all sizes (one recent example includes Microsoft Teams going down for several hours due to an expired cert). Disruptions like this are entirely preventable with proper monitoring in place. In this post, I will go over approaches to monitoring certificate expiration as well as ways to monitor TLS server configuration, using Sensu Go in my examples.

There are many reasons why monitoring certs matters. In addition to causing outages, an expired cert can erode trust in your organization. Perhaps you have to undergo a compliance audit and prove to an auditor that you are secure. Google looks at TLS configuration as part of their search ranking algorithm and having an invalid cert is a great way to get to the bottom of the search results. Keeping your certificates renewed and your server configuration up to date is crucial to running a reliable site. Before we get into it, here’s a quick recap on TLS.

Quick primer on TLS

TLS stands for Transport Layer Security and it’s responsible for encrypting data in transit over the network. In the past, TLS was known as SSL or Secure Socket Layer, but TLS is the modern way of talking about it. TLS performs data encryption, prevents eavesdropping by intermediaries, and allows the client to verify the identity of the server. On the server side there’s a private key and a public certificate that’s been signed by a trusted 3rd party called a Certificate Authority (CA). Certificates are typically valid for up to 2 years although some CAs such as Let’s Encrypt issue ones that are valid for 90 days. Because certificates are issued for a finite period of time it is crucial to monitor the expiration.

Certificate monitoring

There are several approaches to monitoring cert expiration. One of the easiest is to connect to the server and grab the expiration date off the cert and alert when expiration gets close. Hopefully you’re using automation to obtain and renew certs. But even if you’re using a CA like Let’s Encrypt that enables easy automation, it’s still a good practice to monitor expiration because automation can fail or the unexpected can happen. If you have a manual renewal process you’ll want to leave enough time to complete the renewal in time. Some certs have a more stringent approval process and can take longer than you think. I once had a cert that took several weeks worth of paperwork to get it approved and issued.

Let’s start checking the expiration date on my certs using Sensu Go. I’m using the check-ssl-host plugin from sensu-plugins-ssl, part of the Sensu Plugins collection, to do the checking. I’ve set my check to run once a day and I’ll get a warning alert if any cert expires in 30 days and a critical alert if any cert expires in 14 days.

I’m a big fan of visualizing metrics so I’ve created a dashboard to see my certificate status at a glance.

Other monitoring systems have similar checks for certificate expiration and can alert you when a cert approaches expiration. There’s also SaaS services such as Hardenize that specialize in checking your site’s TLS setup.

Server configuration monitoring

TLS server configuration is another thing to pay attention to. Protocols, ciphers, headers, and other server settings all have a bearing on security, performance, and availability. Hopefully you’ve already disabled TLS 1.0 and 1.1 — those early protocols have been deemed insecure and there is almost no reason to have them enabled. If you have any doubt about your TLS config, run a scan so you know where you stand. The SSL Labs test from Qualys is the gold standard in TLS scanning. It checks all aspects of your setup including ciphers, TLS versions, and incorrect certificate chains, will flag any issues, and gives you a grade to motivate you to improve. Any grade less than an “A” means the site has issues that should be addressed.

SSL Labs’ web interface is great for one-off scans but you really want to know right away if your site has issues. For that I recommend running periodic automated scans of your site. Thankfully SSL Labs has an API and we can call it from our monitoring system.

Here I’ve set up a Sensu Go check of my SSL Labs score using the check-ssl-qualys plugin from sensu-plugins-ssl. I run this check once a day and I’ll get an alert if my score drops below an A-.

I’ve set up a dashboard so I can easily see the score of all my sites.

If you want to check a site that isn’t exposed to the internet, you can use the testssl.sh command line tool to scan your internal services. testssl.sh performs nearly the same checks as SSL Labs but can run against any site, internal or external.

If a scan does flag any issues, you’ll want to remediate your config as soon as possible. For that I recommend the TLS Generator from Mozilla. TLS Generator will generate correct configs for popular server software such as nginx and haproxy. There’s a lot of nuance to protocols and cipher specs and getting it wrong could make some people unable to access your site, so pick the settings that work best for your site. It’s a good practice to periodically review your configs as best practices evolve and new protocols come out; for instance, TLS 1.3 is now seeing wider adoption.

Conclusion

It’s important to pay attention to your TLS certificates and server configuration and the best way is through continuous monitoring. Hopefully this post has given you some ideas for how to achieve this.

Have your own tips for monitoring TLS? Share them in our Community Forum!