Your registration has been confirmed. Thank you for signing up!
In this lesson, we will take a deeper look at events in Sensu, create an event manually using common shell tools, and show how events can trigger alerts. This lesson is intended for operators of Sensu, and assumes you have set up a local workshop environment.
In this lesson we will take a deeper look at events in Sensu, create an event manually using common shell tools, and show how events can trigger alerts. This lesson is intended for operators of Sensu, and assumes you have set up a local workshop environment.
We often use the term observability to describe the set of practices around understanding and managing complex software systems. But what do we mean by that?
Observability describes a quality of the system; a measure of how much visibility we have into its state and behaviors. The more you observe, the richer your knowledge is of the system. The richer that knowledge is, the better you can understand and reason about the system.
So what is it that we are observing?
From our vantage point looking at a dashboard, we can only observe what the system shares with us. In traditional software development this involved informational output in the form of logs. Later, some systems were instrumented to allow on-demand inspection.
It’s now commonly understood that there is great value in providing a lot of transparency into system state. We design systems to emit, or allow the inspection of, as much detail as we can. Our data processing and storage systems have evolved beyond the need to constrain or limit this information.
The more of this data we have available, the better off we are when trying to understand unexpected system behaviors.
Modern observability involves two categories of system information; data which is emitted, and data which is queried. Broadly, emitted data is referred to as logs or events, and queried or inspected data is known as metrics or telemetry. These differ primarily in their relationship to time.
Log messages are emitted at irregular and unpredictable times. They are emitted when something happens. Metrics however are queried, either by polling on an periodic basis, or ad hoc when someone is inspecting the system.
The unifying property of these two kinds of information is their timestamp.
Where these two kinds of information differ though, is in structure. Logs are presented as text strings, often in a human language. They are intended to be human-readable and have little to no structure. Metrics are often presented as measurements in numerical form. They are intended to be machine-readable and are often structured as key-value pairs.
For a long time these two kinds of information were handled separately. In Sensu, we have developed a way to bring them together in a unifying structure: the event.
There is a constant stream of observations in different formats coming from every layer of our systems. But we still don’t have great tools for correlating this information. Often what we are looking for is some kind of obvious change of condition, and the way we detect that change is by comparing measurements over time.
But just because two things happened at the same time, doesn’t mean that they are related.
Historically, these two kinds of observation were separate and lacked relational information. Metrics were often loose numerical measurements with no additional context, and logs often appeared out of sequence, making it difficult to group them into meaningful units of work. There was also generally little forethought put into making sure that the system provided complete, accurate, and actionable information.
In Sensu, we have solved part of this problem by unifying all observations into the event data structure.
Sensu operators work with different kinds of observations in the same cognitive space, to find meaningful relationships, define operating thresholds, predict system behaviors, and to discover the meaning within the data.
Sensu events are structured data which include:
timestampentity name indicating the origin (e.g. a server, cloud compute instance, container, or service)metrics structure containing one or more pointsmetadata structure containing key-value pairs, labels, and annotationsid) used to trace event processing{
"timestamp": 1234567890,
"entity": {},
"check": {},
"metrics": {},
"metadata": {
"labels": {},
"annotations": {}
},
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
This generic container provides a tremendous amount of flexibility in terms of the types of observations that can be collected. For all the details on what kind of data events can hold, read the Events reference documentation.
Let’s try making an event manually, using some common command-line tools.
You want to explore the Events API, or want to generate an event manually using common shell tools.
The Sensu API has an /events endpoint which can be used to POST an event to the backend.
Events sent to this API are processed in the same way as events generated by agents.
We can manually create an event using common shell tools like curl or PowerShell’s Invoke-RestMethod cmdlet.
Create an Event Using the Events API.
Any tool that can POST to an HTTP endpoint can be used.
For example, you could instrument your application to emit events directly to Sensu, using standard libraries for HTTP and JSON.
For this exercise we will use curl on Mac/Linux and PowerShell’s Invoke-RestMethod cmdlet on Windows.
Mac and Linux:
curl -i -X POST -H "Authorization: Key ${SENSU_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":2,"output":"ERROR: failed to connect to database."}}' \
"${SENSU_API_URL:-http://127.0.0.1:8080}/api/core/v2/namespaces/${SENSU_NAMESPACE:-default}/events"
NOTE: The
curlcommands included in this lesson should generate output that starts withHTTP/1.1 200 OK,HTTP/1.1 201 Created, orHTTP/1.1 202 Accepted. If you do not see this output, or if you received an error message, please ensure that you completed all of the steps in Lesson2.
Windows (PowerShell):
Invoke-RestMethod `
-Method POST `
-Headers @{"Authorization" = "Key ${Env:SENSU_API_KEY}";} `
-ContentType "application/json" `
-Body '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":2,"output":"ERROR: failed to connect to database."}}' `
-Uri "${Env:SENSU_API_URL}/api/core/v2/namespaces/${Env:SENSU_NAMESPACE}/events"
Let’s break down what we just did there.
We POSTed an event payload, in JSON format, to a secure endpoint in the Events API.
The destination URL was http://127.0.0.1:8080/api/core/v2/namespaces/default/events.
Let’s break this URL down into its parts:
http://127.0.0.1:8080: The address and port of the backend.
api/core/v2: The base address of the Sensu API.
Notice the version in the path. This is how Sensu guarantees to never break backwards compatibility!
namespaces/default: The namespace to POST the event to.
You are likely using the default namespace, but just in case, the command checks to see if the $SENSU_NAMESPACE environment variable is set.
events: This is the root of the Events API.
The API key that we generated in Lesson 3 was used to authenticate the request.
This is stored in the $SENSU_API_KEY environment variable.
The command included an Authorization header, with a value using the format of Key <api_key>.
The payload included a JSON formatted event structure in the POST body.
This event is mimicing the kind of event that an automated check would generate.
{
"entity": {
"metadata":{
"name": "i-424242"
}
},
"check":{
"metadata":{
"name": "my-app"
},
"interval": 30,
"status": 2,
"output": "ERROR: failed to connect to database."
}
}
This event contains an entity with a metadata property, indicating that i-424242 is the name of the node the event came from.
There is also a check structure describing the check that generated it.
The check is named my-app and its interval says that it runs every 30 seconds.
The remaining two are the status and output.
Following common UNIX conventions, if a check executable exits with a status other than 0, it is considered an error.
In such cases, the output is captured and included with the event.
The output message indicates a database connection failure.
At minimum, the backend will store the most recent event for an entity, so we can inspect it via sensuctl.
View a List of Events
sensuctl event list
Example Output:
Entity Check Output Status Silenced Timestamp UUID
──────────────── ──────────── ─────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
learn.sensu.io helloworld Hello, workshop world. 1 false 2021-03-09 22:44:28 -0800 PST 8f0dfc70-8730-4b62-8f16-e4d8673f311f
i-424242 my-app ERROR: failed to connect to database. 2 false 2021-03-10 15:58:25 -0800 PST 0784e60b-96b1-4226-a151-13a645abdf67
But what about the handler we configured in Lesson 4? If you expected that Sensu would process this event using that handler, you might have noticed that nothing happened.
NEXT: Let’s move on to the next exercise to see how event handling works in practice.
You want to create an event that triggers an alert via a handler.
A check event needs to specify which handlers the event should be passed to.
This is part of the check configuration, which is included in the event context via the handlers property.
The backend will match this to the corresponding handler configurations.
Let’s create an event that will be processed using the Mattermost handler we configured in Lesson 4.
Create an Event that Alerts in Mattermost.
This step is the same as last time, but now includes a handlers property in the check structure.
The value is a simple list of handler names.
The Mattermost handler is named mattermost.
{
"entity": {
"metadata":{
"name": "i-424242"
}
},
"check":{
"metadata":{
"name": "my-app"
},
"interval": 30,
"status": 2,
"output": "ERROR : failed to connect to database.",
"handlers": ["mattermost"]
}
}
Run the following command to create the event.
Mac and Linux:
curl \
-i -X POST -H "Authorization: Key ${SENSU_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":2,"output":"ERROR: failed to connect to database.","handlers":["mattermost"]}}' \
"${SENSU_API_URL:-http://127.0.0.1:8080}/api/core/v2/namespaces/${SENSU_NAMESPACE:-default}/events"
Windows (PowerShell):
Invoke-RestMethod `
-Method POST `
-Headers @{"Authorization" = "Key ${Env:SENSU_API_KEY}";} `
-ContentType "application/json" `
-Body '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":2,"output":"ERROR: failed to connect to database.","handlers":["mattermost"]}}' `
-Uri "${Env:SENSU_API_URL}/api/core/v2/namespaces/${Env:SENSU_NAMESPACE}/events"
This time, because this event defines handlers, the backend will generate an alert in Mattermost.
To view the alert, open up Mattermost and login using the default credentials (username: sensu, password: sensu).
Send an Event with a Successful Exit Status.
The last event we sent had an error state. In most scenarios, this would open an incident. Let’s send one more event to indicate that our app is now restored to a functional state.
This time the status will be set to 0 and the output message is 200 OK.
{
"entity": {
"metadata":{
"name": "i-424242"
}
},
"check":{
"metadata":{
"name": "my-app"
},
"interval": 30,
"status": 0,
"output": "200 OK",
"handlers": ["mattermost"]
}
}
Run the following command to create the event.
Mac and Linux:
curl \
-i -X POST -H "Authorization: Key ${SENSU_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":0,"output":"200 OK","handlers":["mattermost"]}}' \
"${SENSU_API_URL:-http://127.0.0.1:8080}/api/core/v2/namespaces/${SENSU_NAMESPACE:-default}/events"
Windows (PowerShell):
Invoke-RestMethod `
-Method POST `
-Headers @{"Authorization" = "Key ${Env:SENSU_API_KEY}";} `
-ContentType "application/json" `
-Body '{"entity":{"metadata":{"name":"i-424242"}},"check":{"metadata":{"name":"my-app"},"interval":30,"status":0,"output":"200 OK","handlers":["mattermost"]}}' `
-Uri "${Env:SENSU_API_URL}/api/core/v2/namespaces/${Env:SENSU_NAMESPACE}/events"
NEXT: Did the handler send messages to Mattermost? If so you’re ready to move on to the next lesson!
In this lesson you learned about the event data structure, and how it unifies all kinds of information into a single data type. We manually created some events with different statuses and learned how to configure events to get sent to the right handlers.
At the moment, our handler will just post every message that comes in directly to the channel. In the next lesson, we will use filters to show how the handler can notice this status change and resolve the incident, while also preventing the chat from being spammed with redundant messages.
Some common use cases for events:
We showed a simple example of a service health check in this exercise, and touched on metrics in the previous exercise. Both topics will be covered in more detail in Lesson 8: Introduction to Checks.
A dead man’s switch can alert on systems that have become unresponsive or are failing to report their status. These are covered in more detail in Lesson 7: Introduction to Agents & Entities, and check TTLs, which are the underlying mechanism used for this, are covered in more detail in Lesson 8: Introduction to Checks.
Events may reference an entity that does not exist in Sensu.
When a backend processes an event that references an entity that is not known, a proxy entity will be created.
Entities and proxy entities are covered in more detail in Lesson 7: Introduction to Agents & Entities and Lesson 12: Introduction to Proxy Entities & Proxy Checks.
Made with #monitoringlove by SensuTM in Canada 🇨🇦 and the USA 🇺🇸. © 2017-2021 – Privacy Statement – Terms & Conditions