Monitoring as code for modern DevOps teams

Software engineering teams that adopt “as-code” practices, like using configuration files and automated workflows instead of manual configuration and tools, gain major improvements in velocity. But even companies that enjoy the success of as-code practices for development and delivery lag behind in applying them to operational concerns like monitoring and observability. Sensu brings monitoring into the as-code fold and allows you to codify complex business requirements into repeatable steps.

The earliest glimmers of monitoring as code started in 1991 with continuous integration (CI): merging all development work to a shared “main” branch multiple times per day ^[1]. In 2009, John Allspaw gave a talk titled “10+ deploys per day” and described how dev and ops teams could achieve faster, more reliable, and more frequent software releases. These cooperative methods, referred to as continuous development (CD), were a natural partner to CI. Ideas about CI/CD spread quickly, and software companies increasingly automated parts of the release and deploy processes.

By 2010, with CI/CD entrenched throughout the industry, it was becoming common to automate setup and configuration with code. This practice, infrastructure as code, proved a great way to boost system reliability, repeatability, and maintainability as well as the speed of provisioning and configuration. Infrastructure as code became a foundational element of DevOps and led to the operating principle called “everything as code,” in which teams pushed to use as-code approaches to automate all aspects of software development and delivery. Today, everything as code is a key strategy in industry-leading organizations’ digital transformation initiatives.

The current challenge facing the software industry is that development and delivery are automated while monitoring and observability are often left out of as-code efforts. IT operations lack adequate tooling to adopt as-code initiatives and thus haven’t enjoyed the same velocity improvements as their software engineering counterparts. The gap only grows as time passes. Monitoring and observability need to catch up to development and delivery in the as-code arena. This is where monitoring as code comes into play.

Benefits of monitoring as code

Companies use as-code practices to reduce the time-to-value for many different facets of IT operations, and now they can use Sensu’s monitoring as code solution to close the gap for monitoring and observability too.

Monitoring as code is an organizational initiative that can be adopted in stages. Start with a single interface or tool and roll out as implementation becomes easier over time.

Organizations that implement monitoring as code can manage their monitoring directly alongside all their other automated processes. Monitoring no longer lives as a separate concern from deployment, but is synchronized with infrastructure and application changes. Monitoring as code provides a unified workflow for IT innovation and IT operations.

Unifying across the entire organization improves communication among teams and overall velocity, with specific advantages like increased productivity through automation, consistent monitoring setup and configuration, and reduced risk of human error. Teams can use existing skills and tools, and familiar workflows and tooling help ease onboarding challenges. And monitoring as code codifies the tribal knowledge often held by only a few.

Infrastructure as code vs monitoring as code

At this point, you might want to know how infrastructure as code differs from monitoring as code. Both offer support for managing resources and dependencies as code. Both push those code configurations to APIs that bring the corresponding systems into their desired state.

In the simplest case, infrastructure as code could be used to automate the installation of monitoring components like agents, scrapers, and plugins. But using infrastructure as code to install these components is only part of the story. Monitoring as code is much more than a tool  to set up your monitoring infrastructure.

Implement monitoring as code: People + Process + Tools

Like many initiatives, monitoring as code requires people working in organizational alignment, a process of repeatable workflows, and a facilitating tool like the Sensu observability pipeline.

People

A successful monitoring as code initiative starts with inviting everyone to get involved — development, operations, and security. Monitoring as code reuses as-code skills and familiar CI/CD workflows and toolchains, which means everyone is qualified!

Introduce the approach among a few engineers, and onboard others over time. Build on the existing familiarity with your current toolchain to encourage organic coordination throughout your organization. As monitoring as code spreads throughout your organization, people will align around their shared interest in the performance and reliability of your systems.

Process

Your monitoring as code process allows teams to rally around a unified workflow for immediate results. IT teams can shift their focus to getting better alerts that are 100% actionable, managed as revision-controlled code changes and new releases, instead of setting goals for reducing noise and getting fewer alerts. Engineering resources can focus on innovation that drives top-line growth rather than avoiding downtime.

Follow a four-stage process: integration, prioritization, acceleration, and innovation.

Integration includes setup time, when you integrate the observability pipeline into your existing as-code tooling like cloud APIs, compute APIs, secrets management, and certificate infrastructure. Sensu’s observability pipeline comes with integrations that are ready-made for coordinating with different systems.

Standardization means identifying existing tools that you can consolidate, like Nagios-style monitoring scripts, StatsD applications, /healthz API endpoints, Prometheus exporters, and Telegraf metrics. Start with the interface or tool that’s the most widely adopted and work team by team to standardize monitoring as code workflows.

Acceleration is the point when you can package common monitoring and observability business requirements as declarative templates. Each template you add becomes available to other teams that use the same tools and services in their app stacks, accelerating adoption across business units.

Innovation is ongoing: it requires iterating on the templates you rolled out in the acceleration stage. The entire app lifecycle is managed as code, from creation to modification to ongoing operations. Any improvements anywhere within the organization are code changes within the unified as-code workflow.

Tools

Monitoring as code requires a dedicated pipeline for ongoing system observation. Sensu’s observability pipeline integrates with your CI/CD pipeline via configuration management. Once deployed, the Sensu observability pipeline provides common capabilities like transforming, standardizing, and routing observability event data. In addition to general monitoring functions, Sensu’s observability pipeline satisfies business requirements for security, reliability, and resiliency with built-in features like single sign-on (SSO) authentication, encryption, autologging, scalability, fault tolerance, and intermittent connectivity.

Sensu offers what monitoring as code really requires:

• Instrumentation for monitoring infrastructure with plugins.
• Scheduling to automatically collect data with subscriptions.
• Diagnosis with check hooks that add context and enrich event data.
• Detection with event filters that filter, deduplicate, and correlate data.
• Notification with handlers that automatically send alerts and create and resolve incidents.
• Processing by routing events and metrics to data storage and analysis platforms like Sumo Logic, Elasticsearch, and InfluxDB.
• Automation with codified remediation actions via integration with runbook automation platforms like Ansible, Rundeck, and Saltstack.

In Sensu’s hands, monitoring as code is more than just collecting data. It’s a complete, end-to-end solution that covers every element of monitoring and observability in a portable, declarative template: a single YAML file.

Sensu’s observability pipeline is the missing link in the evolution of monitoring as code. It includes all the tools you need so that you can focus on extracting value from your monitoring and observability investments and restoring lost velocity throughout IT operations.

Next steps: Take the Sensu Go workshop!

Take the next step in your as-code development! Follow our self-guided Sensu Go workshop for practical lessons about each element in the Sensu observability pipeline, and take your first steps toward implementing monitoring as code for your organization.

Sensu’s monitoring as code solution is available to all Sumo Logic users — download a commercial package at https://sensu.io/downloads and try Sensu for free in our 30-day unlimited use trial.

Read Monitoring as code with Sensu Go and SensuFlow for a deep dive into our git-based, prescriptive monitoring as code workflow that synchronizes your monitoring and observability code with your Sensu deployments.

And don’t forget to join the Sensu Community Forum on Discourse, where you can learn from and share information with other Sensu users!