The Sensu Agent is a lightweight observability client that runs on your infrastructure. Sensu Agents are represented in the Sensu API as Sensu Entities. The primary function of the Sensu Agent is to generate events (observability data) for processing in the Sensu observability pipeline.
The Sensu Agent is a lightweight observability client that runs on your infrastructure. Sensu Agents are represented in the Sensu API as Sensu Entities. The primary function of the Sensu Agent is to generate events (observability data) for processing in the Sensu observability pipeline.
Sensu Entities are API resources that represent anything from a server, compute instance, container/pod, connected device (IoT gateways and devices), network device, application, or even a function. Valid Sensu Entities must have an entity_class
, the most common of which are “agent” and “proxy” – a generic entity class designation for any resource that is not actively under management by a Sensu Agent.
The Sensu Agent is distributed as a single statically compiled binary (sensu-agent
), typically installed via installer packages or Docker containers. All Sensu configuration is loaded at sensu-agent
start time, so the agent must be restarted to update the configuration. The Sensu Agent can be configured via command flags (e.g. sensu-agent start --backend-url
), a config file (see below for example), or environment variables (e.g. SENSU_BACKEND_URL
). If configuration values are set in multiple places, the sensu-agent
will apply configuration in the following order:
Example sensu-agent start --help
configuration flags:
start the sensu agent
Usage:
sensu-agent start [flags]
Flags:
--agent-managed-entity manage this entity via the agent
--allow-list string path to agent execution allow list configuration file
--annotations stringToString entity annotations map (default [])
--api-host string address to bind the Sensu client HTTP API to (default "127.0.0.1")
--api-port int port the Sensu client HTTP API listens on (default 3031)
--assets-burst-limit int asset fetch burst limit (default 100)
--assets-rate-limit float maximum number of assets fetched per second
--backend-handshake-timeout int number of seconds the agent should wait when negotiating a new WebSocket connection (default 15)
--backend-heartbeat-interval int interval at which the agent should send heartbeats to the backend (default 30)
--backend-heartbeat-timeout int number of seconds the agent should wait for a response to a hearbeat (default 45)
--backend-url strings ws/wss URL of Sensu backend server (to specify multiple backends use this flag multiple times) (default [ws://127.0.0.1:8081])
--cache-dir string path to store cached data (default "/var/cache/sensu/sensu-agent")
--cert-file string TLS certificate in PEM format
-c, --config-file string path to sensu-agent config file
--deregister ephemeral agent
--deregistration-handler string deregistration handler that should process the entity deregistration event.
--detect-cloud-provider enable cloud provider detection mechanisms
--disable-assets disable check assets on this agent
--disable-api disable the Agent HTTP API
--disable-sockets disable the Agent TCP and UDP event sockets
--discover-processes indicates whether process discovery should be enabled
--events-burst-limit /events api burst limit
--events-rate-limit maximum number of events transmitted to the backend through the /events api
-h, --help help for start
--insecure-skip-tls-verify skip ssl verification
--keepalive-critical-timeout uint32 number of seconds until agent is considered dead by backend to create a critical event (default 0)
--keepalive-handlers string comma-delimited list of keepalive handlers for this entity. This flag can also be invoked multiple times
--keepalive-interval uint32 number of seconds to send between keepalive events (default 20)
--keepalive-warning-timeout uint32 number of seconds until agent is considered dead by backend to create a warning event (default 120)
--key-file string TLS certificate key in PEM format
--labels stringToString entity labels map (default [])
--log-level string logging level [panic, fatal, error, warn, info, debug] (default "info")
--name string agent name (defaults to hostname) (default "my-hostname")
--namespace string agent namespace (default "default")
--password string agent password (default "P@ssw0rd!")
--redact string comma-delimited customized list of fields to redact
--require-fips indicates whether fips support should be required in openssl
--require-openssl indicates whether openssl should be required instead of go's built-in crypto
--socket-host string address to bind the Sensu client socket to (default "127.0.0.1")
--socket-port int port the Sensu client socket listens on (default 3030)
--statsd-disable disables the statsd listener and metrics server
--statsd-event-handlers strings comma-delimited list of event handlers for statsd metrics
--statsd-flush-interval int number of seconds between statsd flush (default 10)
--statsd-metrics-host string address used for the statsd metrics server (default "127.0.0.1")
--statsd-metrics-port int port used for the statsd metrics server (default 8125)
--subscriptions string comma-delimited list of agent subscriptions
--trusted-ca-file string tls certificate authority
--user string agent user (default "agent")
All of the sensu-agent
command flags (except --config-file
) can be set via the Sensu Agent configuration file using the same flag name (sans the --
).
Example agent.yml
file:
---
# Sensu agent configuration
##
# agent overview
##
#name: "hostname"
#namespace: "default"
#subscriptions:
# - example
#labels:
# example_key: "example value"
#annotations:
# example/key: "example value"
##
# agent configuration
##
#backend-url:
# - "ws://127.0.0.1:8081"
#cache-dir: "/var/cache/sensu/sensu-agent"
#config-file: "/etc/sensu/agent.yml"
#log-level: "warn" # available log levels: panic, fatal, error, warn, info, debug
##
# api configuration
##
#api-host: "127.0.0.1"
#api-port: 3031
#disable-api: false
#events-burst-limit: 10
#events-rate-limit: 10.0
##
# authentication configuration
##
#user: "agent"
#password: "P@ssw0rd!"
##
# monitoring configuration
##
#deregister: false
#deregistration-handler: "example_handler"
#keepalive-timeout: 120
#keepalive-interval: 20
##
# security configuration
##
#insecure-skip-tls-verify: false
#redact:
# - password
# - passwd
# - pass
# - api_key
# - api_token
# - access_key
# - secret_key
# - private_key
# - secret
#trusted-ca-file: "/path/to/trusted-certificate-authorities.pem"
##
# socket configuration
##
#disable-sockets: false
#socket-host: "127.0.0.1"
#socket-port: 3030
##
# statsd configuration
##
#statsd-disable: false
#statsd-event-handlers:
# - example_handler
#statsd-flush-interval: 10
#statsd-metrics-host: "127.0.0.1"
#statsd-metrics-port: 8125
Every sensu-agent
configuration flag (e.g. --backend-url=ws://127.0.0.1:8081
) also has a corresponding environment variable (e.g. SENSU_BACKEND_URL="ws://127.0.0.1:8081"
). All Sensu environment variable names are prefixed with SENSU_
, followed by the corresponding flag in capitalized letters and underscores (_
) instead of dashes (-
). For example, the environment variable for the flag --api-host
is SENSU_API_HOST
.
Example configuration environment variables:
export SENSU_BACKEND_URL="ws://sensu-backend-1:8081 ws://sensu-backend-2:8081 ws://sensu-backend-3:8081"
export SENSU_NAMESPACE="default"
export SENSU_SUBSCRIPTIONS="linux nginx postgres"
export SENSU_USER="agent"
export SENSU_PASSWORD="topsecret"
export SENSU_TRUSTED_CA_FILE="/path/to/ca.pem
Sensu Agents connect to Sensu Backends over a persistent WebSocket (ws
) or encrypted WebSocket Secure (wss
) connection. For optimal network throughput, agents will attempt to negotiate the use of Protobuf serialization when communicating with a Sensu backend that supports it. This communication is via clear text by default.
Agent authentication is required connect to Sensu Backends. The Sensu Agent supports basic authentication (username/password) or mutual transport layer security (mTLS) certificate authentication.
Sensu uses a heartbeat mechanism called “keepalives” to monitor Sensu Agent connectivity. Under the covers, a keepalive
is effectively just another Sensu Event that the Sensu Agent publishes once every --keepalive-interval
configured seconds. The keepalive
event contains the current Sensu Agent configuration and entity properties. If a Sensu Agent fails to report a keepalive
event within the --keepalive-warning-timeout
or --keepalive-critical-timeout
configured thresholds, a warning or critical event is produced on behalf of the Agent.
Keepalive monitoring can be disabled using the --deregister true
flag, which prompts the Sensu backend to remove Sensu Agent entities that have stopped generating keepalive
events.
Sensu uses the publish/subscribe model of communication. Sensu “subscriptions” are equivalent to topics in a traditional publish/subscribe message bus. Sensu backends “publish” requests for observability data and agents who are subscribed to the corresponding topics receive the published request, perform the corresponding monitoring job, and sending the corresponding event data to the observability pipeline.
The publish/subscribe model is powerful in ephemeral or elastic infrastructures, where endpoint identifiers are unpredictable and break traditional host-based monitoring configuration. Instead of configuring monitoring on a per-host basis, Sensu follows a service-based monitoring configuration model where monitors are configured per service topic (e.g. “postgres”), and agents deployed on hosts running the corresponding services simply subscribe to those same topics.
Sensu Entities are structured like other Sensu API resources, with the same common top-level fields/objects (i.e. type
, api_version
, metadata
, spec
). The Sensu Entity spec
object contains the following fields:
Sensu Entity spec
properties:
{
"entity_class": "agent",
"system": {
"hostname": "server-01",
"os": "linux",
"platform": "alpine",
"platform_family": "alpine",
"platform_version": "3.12.6",
"network": {
"interfaces": [
{
"name": "lo",
"addresses": [
"127.0.0.1/8"
]
},
{
"name": "eth0",
"mac": "02:42:ac:13:00:07",
"addresses": [
"172.19.0.7/16"
]
}
]
},
"arch": "amd64",
"libc_type": "musl",
"vm_system": "docker",
"vm_role": "guest",
"cloud_provider": "EC2"
},
"subscriptions": [
"system/linux",
"workshop",
"devel",
"entity:server-01"
],
"last_seen": 1617835646,
"deregister": true,
"deregistration": {},
"user": "agent",
"redact": [
"password",
"passwd",
"pass",
"api_key",
"api_token",
"access_key",
"secret_key",
"private_key",
"secret"
],
"sensu_agent_version": "6.2.7"
}
The only required entity spec
property is entity_class
– all other properties are optional.
NOTE: Sensu Entities that are not associated with a running Sensu Agent are generally referred to as “proxy entities” and will have an
entity_class
of “proxy”. For more information about proxy entities and how they are managed by Sensu, please see Lesson 13: Introduction to Proxy Entities & Proxy Checks.
If you look at your Sensu entity list you’ll note that you already have at least one entity (including one named “i-424242”). Sensu automatically created this entity when we published our first event data to the pipeline, but it isn’t associated with a running agent, so its agent class is set to “proxy”.
$ sensuctl entity list
ID Class OS Subscriptions Last Seen
────────── ─────── ──── ─────────────── ───────────
i-424242 proxy N/A
There are two types of entity management in Sensu Go 6.x:
API-managed entities (cloud-native).
Certain entity properties can be modified in real-time via the API, CLI, or web app. Agent entities are API-managed by default in Sensu Go 6.x (API management was not available for agent entities in Sensu Go 5.x).
Agent-managed entities (traditional).
All entity properties are managed via sensu-agent
configuration (command flags, config file, or environment variables). Agent-managed entity updates are applied by modifying one or more configuration attributes and restarting the sensu-agent
process. To enable agent management for an agent entity in Sensu Go 6.x, set the --agent-managed-entity
flag.
NOTE: All agent entities are created using the sensu-agent
configuration; i.e. the initial configuration for API-managed entities is provided by the sensu-agent
, but subsequent edits made to the sensu-agent
(via config flags, config file, or environment variables) will be ignored unless the --agent-managed-entity
flag is set.
A “proxy entity” is any Sensu entity resource that is not actively under management by a Sensu Agent. Proxy entities have an entity_class
of “proxy” and can be used to represent any resource under management by Sensu, though proxy entities are generally used to represent resources where we can’t run an agent (e.g. certain IoT/connected devices, or network devices), or resources that may emit observability data directly to Sensu without the need for an agent (e.g. applications or serverless functions).
NOTE: For more information about proxy entities and how they are managed by Sensu, please see Lesson 13: Introduction to Proxy Entities & Proxy Checks.
The Sensu Agent Events API provides an HTTP POST endpoint for publishing observabilty data to the Sensu Observability Pipeline. While similar to the Sensu backend Events API, the Sensu Agent Events API differs in a few ways:
entity
object (all events are automatically associated with the agent entity)The agent places events created via the /events POST
endpoint into a durable queue stored on disk. In case of a loss of connection with the backend or agent shutdown, the agent preserves queued event data. When the connection is reestablished, the agent sends the queued events to the backend.
To learn more about the Agent Events API, please visit the Sensu Agent reference documentation.
The Sensu Agent Events API makes it easy to implement dead mans switches with as little as one line of bash
or Powershell (see below for examples). The Sensu Event specification supports an event.check.ttl
attribute which can be set to instruct the Sensu platform to expect subsequent event updates; if another event is not received within the configured TTL interval, Sensu generates a TTL event with a status like “Last check execution was 120 seconds ago” and processes the event using the configured handlers.
Dead mans switches are useful for monitoring jobs like nighly backup jobs (e.g. a bash script scheduled via a cron job). A simple one-liner at the end of the backup script can be used to report on the backup status (e.g. "Backup completed successfully. Backup data is available at <backup location URI>.
) with a ~25 hour TTL to account for long running backup jobs. A failed backup job will result in a TTL event without the need for any if/then/else conditional logic in the script (i.e. no need to also send an event if the script fails) – the mere absence of an “OK” event is all that is needed.
Bash
curl -XPOST -H 'Content-Type: application/json' -d '{"check":{"metadata":{"name":"dead-mans-switch"},"output":"Alert if another event is not received in 30s","status":0,"ttl":30}}' 127.0.0.1:3031/events
Powershell
Invoke-RestMethod -Method POST -ContentType "application/json" -Body '{"check":{"metadata":{"name":"dead-mans-switch"},"output":"Alert if another event is not received in 30s","status":0,"ttl":30}}' -Uri "${Env:SENSU_API_URL}/api/core/v2/namespaces/${Env:SENSU_NAMESPACE}/events"
Create a proxy Entity using the Sensu Entities API.
In Sensu, any entity that does not under active management by a Sensu Agent is considered a “proxy” entity. Let’s create a proxy entity as a precursor to installing our first agent so we can better understand the association between the agent (a software component) and its entity (a Sensu API resource).
Mac and Linux users:
curl -i -X PUT -H "Authorization: Key ${SENSU_API_KEY}"\
-H "Content-Type: application/json"\
-d "{\"entity_class\": \"proxy\", \"metadata\":{\"name\":\"i-424242\",\"namespace\":\"${SENSU_NAMESPACE:-default}\"}}"\
"${SENSU_API_URL:-http://127.0.0.1:8080}/api/core/v2/namespaces/${SENSU_NAMESPACE:-default}/entities/i-424242"
Windows users (Powershell):
Invoke-RestMethod `
-Method PUT `
-Headers @{"Authorization" = "Key ${Env:SENSU_API_KEY}";} `
-ContentType "application/json" `
-Body "{`"entity_class`": `"proxy`", `"metadata`":{`"name`":`"i-424242`",`"namespace`":`"${Env:SENSU_NAMESPACE}`"}}" `
-Uri "${Env:SENSU_API_URL}/api/core/v2/namespaces/${Env:SENSU_NAMESPACE}/entities/i-424242"
Do you see a new entity in Sensu? The Sensu Entities API enables us to register endpoints for management by Sensu using external mechanisms (e.g. third-party integrations, discovery scripts, etc). Try deleting the entity (e.g. via sensuctl entity delete i-424242
or using the Sensu web app), and recreating it with different properties.
NEXT: If you see the example proxy entity in Sensu, you’re ready to move on to the next exercise.
The Sensu Agent is available for Docker, Ubuntu/Debian, RHEL/CentOS, Windows, MacOS, and FreeBSD. This exercise will focus on a simplified install for running a Sensu agent on your local workstation, but it will not go into as much detail as the official documentation. The official Sensu Go installation documentation provides detailed instructions for installing & operating Sensu Agents.
Configure environment variables.
NOTE: instructor-led workshop users may need to download an .envrc
or .envrc.ps1
file at this time (if they haven’t already); please consult SETUP.md for more information.
Mac and Linux users (.envrc
):
source .envrc
env | grep SENSU
Windows users (.envrc.ps1
):
. .\.envrc.ps1
Get-ChildItem env: | Out-String -Stream | Select-String -Pattern SENSU
The output should include the expected values for SENSU_VERSION
(i.e. a supported Sensu version such as “6.2.7”), SENSU_BUILD
(i.e. a supported Sensu build such as “4449”), SENSU_BACKEND_URL
, SENSU_NAMESPACE
, SENSU_USER
, and SENSU_PASSWORD
.
Download & install the latest Sensu Agent for MacOS, Windows, or Linux.
Mac users:
curl -LO https://s3-us-west-2.amazonaws.com/sensu.io/sensu-go/${SENSU_VERSION}/sensu-go_${SENSU_VERSION}_darwin_amd64.tar.gz
tar -xzf sensu-go_${SENSU_VERSION}_darwin_amd64.tar.gz sensu-agent
rm sensu-go_${SENSU_VERSION}_darwin_amd64.tar.gz
sudo mkdir -p /usr/local/bin/
sudo mv sensu-agent /usr/local/bin/sensu-agent
Windows users (Powershell):
First download the Sensu agent package:
Invoke-WebRequest `
-Uri "https://s3-us-west-2.amazonaws.com/sensu.io/sensu-go/${Env:SENSU_VERSION}/sensu-go-agent_${Env:SENSU_VERSION}.${Env:SENSU_BUILD}_en-US.x64.msi" `
-OutFile "${Env:UserProfile}\sensu-go-agent_${Env:SENSU_VERSION}.${Env:SENSU_BUILD}_en-US.x64.msi"
Once the download is complete, install the package and add the executable into your command path:
msiexec.exe /i "${Env:UserProfile}\sensu-go-agent_${Env:SENSU_VERSION}.${Env:SENSU_BUILD}_en-US.x64.msi" /qr
rm "${Env:UserProfile}\sensu-go-agent_${Env:SENSU_VERSION}.${Env:SENSU_BUILD}_en-US.x64.msi"
${Env:Path} += ";C:\Program Files\Sensu\sensu-agent\bin"
Linux users (amd64):
curl -LO https://s3-us-west-2.amazonaws.com/sensu.io/sensu-go/${SENSU_VERSION}/sensu-go_${SENSU_VERSION}_linux_amd64.tar.gz
sudo -E tar -xzf sensu-go_${SENSU_VERSION}_linux_amd64.tar.gz -C /usr/bin/
rm sensu-go_${SENSU_VERSION}_linux_amd64.tar.gz
Linux users (arm64):
curl -LO https://s3-us-west-2.amazonaws.com/sensu.io/sensu-go/${SENSU_VERSION}/sensu-go_${SENSU_VERSION}_linux_arm64.tar.gz
sudo -E tar -xzf sensu-go_${SENSU_VERSION}_linux_arm64.tar.gz -C /usr/bin/
rm sensu-go_${SENSU_VERSION}_linux_arm64.tar.gz
Start the Sensu Agent.
NOTE: Mac and Linux users are encouraged to run the sensu-agent
with a service account (e.g. a sensu
or _sensu
user). To create this service account, please refer to the “Create a Sensu user account (service account)” instructions in TROUBLESHOOTING.md. Alternatively, Mac and Linux users may modify the sensu-agent start
commands by removing the -u sensu
or -u _sensu
flag.
Mac users:
TMPDIR=/opt/sensu/tmp\
sudo -E -u _sensu sensu-agent start\
--name workshop\
--backend-url ${SENSU_BACKEND_URL}\
--namespace ${SENSU_NAMESPACE}\
--subscriptions system/macos,workshop\
--deregister true\
--cache-dir /opt/sensu/sensu-agent/cache\
--user ${SENSU_USER}\
--password ${SENSU_PASSWORD}
Windows users (Powershell):
sensu-agent start `
--name workshop `
--backend-url ${Env:SENSU_BACKEND_URL} `
--namespace ${Env:SENSU_NAMESPACE} `
--subscriptions system/windows,workshop `
--deregister true `
--user ${Env:SENSU_USER} `
--password ${Env:SENSU_PASSWORD}
Linux users:
sudo -E -u sensu sensu-agent start\
--name workshop\
--backend-url ${SENSU_BACKEND_URL}\
--namespace ${SENSU_NAMESPACE}\
--subscriptions system/linux,workshop\
--deregister true\
--user ${SENSU_USER}\
--password ${SENSU_PASSWORD}
Verify that your agent is running and connected to the Sensu Backend by consulting sensuctl
(e.g. sensuctl entity list
) or the Sensu web app.
NEXT: If your Sensu Agent has successfully connected to your backend, you’re ready to move on to the next exercise.
Sensu Agents are represented in the Sensu API as Sensu Entities. As a result, some Sensu Agent configuration parameters are used to manage the behavior of the agent (e.g. TLS certificates, authentication, subscriptions, etc), while others are used to configure the corresponding Sensu Entity (e.g. metadata properties like labels and annotations).
Let’s stop our agent and modify its configuration:
Stop the Sensu Agent.
If you started your agent in the previous exercise using the sensu-agent start
command, you can stop the agent by pressing Control-C
in your terminal.
Configure Sensu Agent.
The Sensu Agent supports configuration via command flags (e.g. --backend-url
), a configuration file (e.g. /etc/sensu/agent.yml
), or environment variables (e.g. SENSU_SUBSCRIPTIONS
). For training purposes we will use a mix of all three, however in practice you may find that just one method is best suited for your environment (e.g. on Kubernetes or other container-based environments it may be easiest to manage all configuration via environment variables).
At minimum, the Sensu Agent requires a Sensu backend URL (a websocket API to connect to), and one or more subscriptions (observability topics the agent will subscribe to). We’ll also add some agent metadata in the form of labels & annotations.
Let’s start by creating an agent.yaml
configuration file in one of the following recommended locations:
/opt/sensu/agent.yaml
${Env:UserProfile}\Sensu\agent.yaml
/etc/sensu/agent.yaml
Now let’s copy the following contents into the agent.yaml
file:
---
backend_url: ws://127.0.0.1:8080
name: workshop
labels:
foo: bar
environment: training
annotations:
sensu.io/plugins/rockerchat/config/alias: sensu-trainee
deregister: true
Make sure to save the contents of the file before moving on to the next step.
Configure environment variables.
NOTE: instructor-led workshop users may need to download an .envrc
or .envrc.ps1
file at this time (if they haven’t already); please consult SETUP.md for more information.
Mac and Linux users (.envrc
):
source .envrc
env | grep SENSU
Windows users (.envrc.ps1
):
. .\.envrc.ps1
Get-ChildItem env: | Out-String -Stream | Select-String -Pattern SENSU
The output should include the expected values for SENSU_USER
and SENSU_PASSWORD
.
Start/Restart the Sensu Agent.
Let’s start/restart the agent from the command line again, this time using a mix of environment variables and our configuration file to configure the agent.
Mac users:
TMPDIR=/opt/sensu/tmp\
SENSU_SUBSCRIPTIONS="system/macos workshop"\
sudo -E sensu-agent start\
--config-file /opt/sensu/agent.yaml\
--cache-dir /opt/sensu/sensu-agent/cache\
--user ${SENSU_USER}\
--password ${SENSU_PASSWORD}
Windows users (Powershell):
${Env:SENSU_SUBSCRIPTIONS}="system/windows workshop" `
sensu-agent start `
--config-file "${Env:UserProfile}\Sensu\agent.yaml" `
--user ${Env:SENSU_USER} `
--password ${Env:SENSU_PASSWORD}
Linux users:
SENSU_SUBSCRIPTIONS="system/linux workshop"\
sudo -E -u sensu sensu-agent start\
--config-file "/etc/sensu/agent.yaml"\
--user ${SENSU_USER}\
--password ${SENSU_PASSWORD}
Notice that we have moved the --name
, --backend-url
, and --deregister
configuration settings into the agent.yaml
config file, and we are now explicitly setting the SENSU_SUBSCRIPTIONS
environment variable in place of --subscriptions
… but how/where is --namespace
being set? The Sensu Agent is reading the value of SENSU_NAMESPACE
from the environment variable, without the need for explicitly setting the variable with the sensu-agent start
command.
In the previous exercise we provided all of the configuration via sensu-agent start
command flags. In this exercise we’ve moved some configuration to a config file, and other configuration to environment variables. Understanding how to configure Sensu using all three methods – config flags, config file, and environment variables – is very useful in heterogeneus environments (e.g. mix of servers, compute instances, and containers) where a configuration method that is easier to manage in one context might not be as easy in another context.