Turn-Key Infrastructure and Application Monitoring

The way businesses obtain infrastructure has changed dramatically over the past decade, as Infrastructure-as-a-Service (IaaS) has taken the place of self-hosted infrastructure for most IT deployments. At the same time, it has become common to build complex infrastructures that blend components from multiple providers – such as two or more public clouds (aka. multicloud infrastructure) or mixing an on-prem data center and a public cloud (aka. hybrid cloud infrastructure).

The practice of monitoring has also evolved. Today, it’s common to talk in terms of “observability,” which implies unified services that correlate logs, metrics and tracing to deliver holistic visibility into complex environments.

Here at Sensu, we are firmly in the “observability” camp, but whichever terminology you use to talk about monitoring, infrastructure monitoring remains absolutely critical. It’s a great way to get ahead of performance or reliability issues that could disrupt workloads. It also plays a central role in controlling costs and detecting security risks.

For all of these reasons, understanding how infrastructure monitoring works in modern IT environments and knowing how to choose the right infrastructure monitoring solution are crucial considerations for any IT organization.

Keep reading for a deep dive into modern infrastructure monitoring, along with tips on using Sensu to make infrastructure monitoring as effective as possible for your organization.

What Is Infrastructure Monitoring?

Infrastructure monitoring is the process of collecting and analyzing data from IT infrastructure, systems, and processes, and using that data to improve business outcomes and drive data value across the whole organization.

Although the specific types of data collected by infrastructure monitoring tools may vary depending on which environment you run, common examples of insights that you can gain through infrastructure monitoring include tracking CPU, memory, disk, and other resources consumed on the physical or virtual compute layer, relative to the total resources available.

By measuring data like this, you can stay on top of infrastructure issues – such as insufficient CPU or memory resources – that, if left unaddressed, could disrupt workloads. With a Sensu-enabled infrastructure, IT professionals can be alerted to these kinds of issues in realtime, allowing the team to make decisions and respond to risk as quickly as possible.

How Infrastructure Monitoring Works

Although all infrastructure monitoring tools and processes focus on collecting the same general types of information, there are many approaches to implementing infrastructure monitoring. The types you use will depend on which types of infrastructure you need to monitor.

For example, if you are monitoring VMs, and you have full control over the host operating system, you can use agent-based infrastructure monitoring. In this case, you’d install software agents on the VMs. Those agents would collect data about CPU, memory, and storage resources on each VM, then send the data to your infrastructure monitoring tools.

In situations where you don’t have full control over your host infrastructure – which could be the case if, for example, you are using a managed Kubernetes service (like AWS EKS) where the provider manages the host servers for you – you may need to use third-party services (like AWS CloudWatch) to collect data about infrastructure health and performance.

Managed services tend also to offer lower fidelity out-of-the-box because they collect metrics at relatively infrequent intervals (such as every five minutes), whereas agent-bast host monitoring solutions can collect hundreds of metrics multiple times a minute.

With Sensu’s large catalog of integrations and flexible architecture, it’s no problem to monitor either scenario. Sensu is equally at home in managed environments or in self-hosted infrastructure. Or, as we see increasingly in practice, a mix of both of these kinds of environments, across multiple datacenters.

Regardless – whether you have full control and ownership over your infrastructure or the infrastructure is provided and managed by a third party, and regardless of how frequently you can pull infrastructure metrics – monitoring the infrastructure is important.

Infrastructure Monitoring Benefits

The main benefit of infrastructure monitoring is that it helps you detect infrastructure problems early-on so that you can fix them before they cause your workloads to fail.

For example, an infrastructure monitoring tool like Sensu might alert you to the fact that the CPU usage on your servers is nearing 100 percent. In that case, you’d probably want to allocate more CPU resources to your infrastructure (by, for example, deploying a new type of VM instance with more CPU, or setting up autoscaling in order to add capacity automatically) in order to avoid a situation where your workloads stop handling requests in a timely fashion because they don’t have enough CPU resources to process them. Even if the underlying cause of the problem lies with your application rather than your infrastructure, making more infrastructure resources available temporarily may provide a stopgap fix that will keep the workload running smoothly until you can address the root cause of the issue.

That highlights another point: figuring out where the problem is can be difficult. With a tool like Sensu, you can monitor your infrastructure and your application at the same time, which will help you locate and address the root cause of the problem. While you may be able to stopgap a problem with extra CPU or memory, that’s a very costly fix, compared to a one-line fix in your application. In Sensu you can see both infrastructure and application monitoring side-by-side in the same tool, and focus your efforts, and your spending, on the right solution.

Indeed, maintaining workload health and performance is not the only benefit of infrastructure monitoring. Infrastructure monitoring is also important for helping to control costs by identifying situations where you have significantly more CPU, memory, or storage resources allocated to your workloads than the workloads are actually consuming. In that case, you could scale back your infrastructure resource allocations in order to reduce your spending.

In multicloud or hybrid cloud environments, infrastructure monitoring can also be helpful for providing guidance about where to place workloads. By tracking which infrastructure resources are available across different clouds or data centers and measuring the performance of all of them in real time, you can make more informed decisions about migrating workloads or data from one part of your infrastructure to another. A platform like Sensu can operate in many different environments and is designed from the beginning to support multi-cloud environments.

Requirements and Challenges

Although the types of data on which infrastructure monitoring focuses is straightforward, infrastructure monitoring can be challenging in modern environments due to the complexity of the underlying infrastructure. Common infrastructure monitoring challenges that businesses are likely to face today include:

• Multicloud monitoring: Collecting infrastructure metrics from across multiple cloud platforms typically requires tools that can ingest and analyze data from multiple cloud services. You may need, for example, to collect data from AWS CloudWatch and Azure Monitor at the same time, then analyze and visualize the data through a central platform to track infrastructure health across your multicloud environment.
• Hybrid cloud monitoring: In a hybrid cloud environment, which combines public cloud infrastructure or services with infrastructure that businesses own and manage themselves, being able to distinguish between each part of the infrastructure is critical. You need to know, for example, whether a failing VM is being managed by a public cloud provider, or whether the VM is running on a server inside your own data center.
• Microservice monitoring: Although monitoring microservices is primarily a form of application monitoring rather than infrastructure monitoring, tracking the health of microservices requires an ability to monitor the container infrastructure that hosts them. You must also be able to map container infrastructure performance problems (such as a failed container) to the specific microservices they impact.

Put simply, modern IT infrastructures are likely to consist of a range of different components. Infrastructure monitoring tools need to be able to collect and analyze data from across all of them. Just as important, monitoring tools must deliver insights that can be mapped back onto the various components of your infrastructure so that you can pinpoint the root cause of problems.

One of the great things about Sensu is that it was built with this understanding in mind, and seamlessly integrates many different environments into a single pane of glass. Also, with full support for role-based access control (RBAC) it is easy to enable large organizations where different groups will need to access different parts of the infrastructure.

Infrastructure Monitoring Best Practices

To make infrastructure monitoring as effective as possible, consider the following best practices:

• Monitor all of your infrastructure: Don’t settle for the low-hanging fruit (like VMs) that is easy to monitor. Ensure that you are monitoring all of your infrastructure resources – containers, serverless functions, databases, and beyond.
• Don’t neglect third-party infrastructure: Along similar lines, resist the temptation to assume that infrastructure managed by someone else – such as a public cloud provider – doesn’t need to be monitored. Even the best-managed public cloud IaaS services can fail due to issues on the provider’s end or to configuration mistakes made by you when deploying the infrastructure.
• Monitor in real time: Collect and analyze infrastructure metrics in real time, or as close to it as you can get. Delays of even just a few minutes may prevent you from detecting emergent performance issues quickly enough to address them before suffering a disruption.
• Consolidate infrastructure monitoring: Rather than relying on multiple tools to expose and analyze infrastructure metrics, consolidate your monitoring around a central platform that can collect data from across the various IaaS platforms, on-prem resources, and other resources you need to track.

How to Choose the Right Infrastructure Monitoring Tools

There are a variety of infrastructure monitoring solutions available today. Some are built into cloud platforms. Others are standalone options that can work across multiple clouds or other types of environment. We happen to think Sensu is the best choice for most scenarios (obviously), but you’ll need to make the decision for yourself.

To select the right infrastructure monitoring tool for your needs, consider factors such as:

• Ease of use: Do you want a fully managed solution that you can deploy quickly on demand? Or are you willing to set up and manage the solution yourself on your own hardware?
• Flexibility and integrations: Unless you have just one cloud or data center, or just one type of infrastructure to monitor, you’ll want a monitoring tool that can work across different types of environments and integrate with infrastructure resources seamlessly.
• Remote management: The ability to deploy infrastructure monitoring software and collect data remotely can help to reduce operational overhead by eliminating the need for complex third-party automation processes or tools. For instance, it saves you from having to deploy separate configuration management tools to set up hundreds or thousands of monitoring agents.
• Cost: When evaluating infrastructure monitoring tool costs, consider not just the direct costs, like licensing fees. You should also factor in data egress and storage costs, which could become significant if you move or store a lot of data as part of your infrastructure monitoring operations. And you should consider the cost of having to build and maintain homegrown infrastructure monitoring solutions (or configure open source solutions to meet your needs), as opposed to licensing a ready-made monitoring platform. In most cases, unless your business’s main business is telemetry, it’s probably cheaper from a TCO perspective to use an existing solution.

Sensu: A unified infrastructure and application monitoring solution

So now that we’ve laid out some considerations and best practices, let’s talk about how Sensu enables each of those points:

Best Practices

• Monitor all of your infrastructure: Sensu is equally at home in a container, a VM, or running on bare metal. Our catalog of integrations already includes every major computing environment whether it’s AWS, GCP, Azure, or Managed Kubernetes. If you do find yourself in an environment where Sensu isn’t already available, it’s very easy to extend it to support new technologies. A few lines of code and you can introduce an entirely new integration and deploy those new capabilities remotely, and without a redeploy.
• Don’t neglect third-party infrastructure: While most people run the Sensu Agent directly on the system they are monitoring, a lesser known use case is to run it as an outside observer. The Sensu Agent can even monitor systems that you don’t have administrative access to, including things like external 3rd party dependencies. The flexibility of Sensu’s API-based pipeline architecture allows you to work with even the most challenging monitoring scenarios.
• Monitor in real time: Sensu can collect, understand, and react to events in your system within milliseconds. That’s due to its highly optimized processing pipeline, and a design that doesn’t require storage to understand and alert on what is happening. Many monitoring tools need to collect data, store it, index it, and then analyze it before a human is alerted. This often takes many minutes to produce actionable insight. In the modern world of digital business, even five minutes of downtime can be devastating to many enterprises. With Sensu you can avoid these dangers, and even build in automatic remediation so that the system can heal itself, without needing a human to be involved at all.
• Consolidate infrastructure monitoring: Sensu brings all your monitoring together into a single pane of glass. No matter where the infrastructure is, or what kind of system it is, you can see it side-by-side with every other component of your system. This also includes application and service monitoring as well as infrastructure. Being able to see everything at once – and make decisions in real-time – gives you a true sense of understanding and control over even very large and complicated systems.

Deciding Factors

• Ease of use: First, let’s talk about Sensu’s weaknesses here. Sensu is not a hosted solution. Unlike some popular monitoring tools Sensu runs on your infrastructure and needs to be managed by an IT professional. However, once installed, Sensu makes it very easy to onboard other parts of the organization without the need for those people to be monitoring professionals. With RBAC and an API-based architecture, with both CLI and web-based tools, and with the ability to change its configuration without the need for a redeploy, Sensu is extraordinarily flexible and easy to use even by non-specialists.
• Flexibility and integrations: As described above, Sensu can operate in any environment, and there are hundreds of integrations available, as well as default configurations following industry-standard best practices, baked directly into our integration catalog. If there’s something missing that you need, it’s very easy to extend Sensu with new functionality. Creating a plugin and deploying it via the API means that anyone can add new behaviors to Sensu without needing to do major lifting or schedule service outages for deployments.

• Remote management: One of Sensu’s strongest features is its API-based architecture. Every function, and every entity within Sensu is available via the API. Sensu ships with both CLI tools and a fully-featured web app, but those tools sit on top of APIs, which can be interacted with by any source – including CI and cron jobs, or other automation systems. Remote management is there by default with Sensu and couldn’t be more flexible or easy to use.
• Cost: Another little known feature of Sensu is how much it can help control the costs of monitoring. First off, we give you access to every paid enterprise feature of the system, for free, on up to 100 nodes. That gives you a huge head start without needing to spend a penny on Sensu. But Sensu can actually cut costs in other ways as well. Sensu’s observability pipeline can understand and process monitoring without the need to store it in a database. That means you can avoid the costly egress and storage fees that other monitoring solutions require. You don’t need to move the data outside of your infrastructure to act on it, and in fact, you can intelligently apply filters which can decide which data to send off to long-term storage, and which data to ignore. This can represent massive cost savings as you narrow in on parts of your system that are immediately actionable. So, rather than costing you money, you could actually make money by using Sensu. That’s not a primary feature of the system, but it’s a pretty happy side effect, right?

Sensu Plus: An Integrated Analytics Solution for All Sensu Users

For Sensu users who want an integrated analytics engine to extract insights from Sensu’s observability pipeline data, we have launched Sensu Plus – a bundled solution that offers the Sumo Logic analytical backend in addition to the Sensu Monitoring as Code solution.

Sumo Logic dashboards provide immediate visibility into Sensu’s real-time inventory data, and overall node health. Sumo Logic’s comprehensive analytics tools make these dashboards interactive, enabling users to drill down from high-level overviews to individual nodes or events.

Get Started with Sensu

So now that we’ve convinced you of the need for infrastructure monitoring, and shown you that Sensu is an excellent choice for that task, we encourage you to give it a try. You can download and install Sensu today, and learn how to best leverage its features in our self-directed Sensu Go Workshop. You don’t need to talk to a salesperson to start using Sensu, but if you have questions feel free to reach out for a no-pressure conversation about how Sensu can elevate your IT operations and provide the flexibility you’ll need to grow in future.