💾 Archived View for perso.pw › blog › articles › reed-alert.gmi captured on 2022-04-29 at 12:31:13. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2021-12-17)

➡️ Next capture (2023-01-29)

-=-=-=-=-=-=-

Monitor your systems with reed-alert

NIL=> Comment on Mastodon

This article will present my software __reed-alert__, it checks

user-defined states and send user-defined notification. I made it

really easy to use but still configurable and extensible.

Description

__reed-alert__ is _not_ a monitoring tool producing graph or storing

values. It does a job sysadmins are looking for because there are no

alternative product (the alternatives comes from a very huge

infrastructure like Zabbix so it's not comparable).

From its configuration file, __reed-alert__ will check various states

and then, if it fails, will trigger a command to send a notification

(totally user-defined).

Fetch it

This is a open-source and free software released under MIT license,

you can install it with the following command:

# git clone git://bitreich.org/reed-alert

# cd reed-alert

# make

# doas make install

This will install a script `reed-alert` in /usr/local/bin/ with the

default Makefile variables. It will try to use ecl and then sbcl if

ecl is not installed.

A __README__ file is available as documentation to describe how to use

it, but we will see here how to get started quickly.

You will find a few files there, __reed-alert__ is a Common LISP

software and it has been chose for (I hope) good reasons that the

configuration file is plain Common LISP.

There is a configuration file looking like a real world example named

named **example.lisp** containing lot of cases.

Let's start

In order to use __reed-alert__ we only need to create a new

configuration file and then add a cron job.

Configuration

We are going to see how to configure __reed-alert__. You can find more

explanations or details in the __README__ file.

Alerts

We have to configure two kind of parameters, first we need to set-up a

way to receive alerts, easiest way to do so is by sending a mail with

"mail" command. Alerts are declared with the function **alert** and as

parameters the alert name and the command to be executed. Some

variables are replaced with values from the probe, in the __README__

file you can find the list of probes, it looks like %date% or

%params%.

In Common LISP functions are called by using a parenthesis before its

name and until the parenthesis is closed, we are giving its

parameters.

Example:

(alert mail "echo 'problem on %hostname%' | mail me@example.com")

One should take care about nesting quotes here.

__reed-alert__ will fork a shell to start the command, so pipes and

redirection works. You can be creative when writing alerts that:

+ use a SMS service

+ write a script to post on a forum

+ publishing a file on a server

+ send text to IRC with ii client

Checks

Now we have some alerts, we will configure some checks in order to

make __reed-alert__ useful. It uses *probes* which are pre-defined

checks with parameters, a probe could be "has this file not been

updated since N minutes ?" or "Is the disk space usage of partition X

more than Y ?"

I chose to name the function "=>" to make a check, it isn't a name

and reminds an item or something going forward. Both previous example

using our previous mail notifier would look like:

(=> mail file-updated :path "/program/file.generated" :limit "10")

(=> mail disk-usage :limit 90)

It's also possible to use shell commands and check the return code

using the __command__ probe, allowing the user to define useful

checks.

(=> mail command :command "echo '/is-this-gopher-server-up?' | nc -w 3 dataswamp.org 70"

:desc "dataswamp.org gopher server")

We use echo + netcat to check if a connection to a socket works. The

"COMMAND".

Garniture

We wrote the minimum required to configure __reed-alert__, now the

configuration file so your **my-config.lisp** file should looks like

this:

(alert mail "echo 'problem on %hostname%' | mail me@example.com")

(=> mail file-updated :path "/program/file.generated" :limit "10")

(=> mail disk-usage :limit 90)

Now, you can start it every 5 minutes from a crontab with this:

*/5 * * * * ( reed-alert /path/to/my-config.lisp )

If you prefer to use ecl:

*/5 * * * * ( reed-alert /path/to/my-config.lisp )

The time between each run is up to you, depending on what you monitor.

Important

By default, when a check returns a failure, __reed-alert__ will only

trigger the notifier associated once it reach the 3rd failure. And

then, will notify again when the service is back (the variable %state%

is replaced by start or end to know if it starts or stops.)

This is to prevent reed-alert to send a notification each time it

checks, there is absolutely no need for this for most users.

The number of failures before triggering can be modified by using the

keyword ":try" as in the following example:

(=> mail disk-usage :limit 90 :try 1)

In this case, you will get notified at the first failure of it.

The number of failures of failed checks is stored in files (1 per

check) in the "states/" directory of reed-alert working directory.