How-to: Nagios MQTT notifications

Introduction

Having got email notifications working from Nagios and being inundated with emails that I will no doubt ignore I got to thinking about how I could automate various other tasks depending on the state of the network.

Nagios “knows” what’s up and what’s not.

I have various devices on my network that can switch things off, switch things on, log events, light up indicators, etc, etc, and the various things that are sensed and the various things that happen each have a corresponding MQTT message – ie all the various systems, whether they measure and detect, make decision or send instructions for some kind of action, talk to each other using MQTT.

It would be ideal, I thought, if Nagios could stop sending emails and simply update my existing system with information relating to the things it knew – ie by sending MQTT messages (or “publishing” as its really called). Once that information is within my wider setup (ie once I have MQTT messages describing Nagios “knowledge” I can do all sorts of other non-Nagios stuff).

This post, therefore, details my findings in the form of a how-to set it up with all the trial-and-error removed.

Assumptions

As I already have various parts of this system running I haven’t documented how to install them – maybe I will in separate posts at some point. But at this time, the following instructions assume that:

• Nagios is running as described here, with
• a basic config something like this, and
• working email notifications as detailed in this post

It also assumes that MQTT is available – a post may follow to detail how. for the time being, its really not difficult and there are loads of resources online to help you do that. I’d suggest following something like this.

How to…

Assuming you have the above running, enabling the publishing of MQTT messages by Nagios is straightforward.

I achieved it by utilising the fact that Nagios already triggers a command when events occur. Each time the status of a host or service changes, Nagios executes the following internal commands:

notify-host-by-email
notify-service-by-email

by amending the external commands Nagios executes through the above internal commands I got an MQTT client to publish a message containing similar information to that sent as an email.

Step-by-step

Install mosquitto tools so as to have access to the single mosquitto_pub command (no need to enable mosquitto as a service – it will be called from the command line each time it is required to publish and note that if an MQTT broker is running locally on the Nagios host then the tools to publish may already exist locally)

sudo apt update
sudo apt install -y mosquitto mosquitto-clients

Test the publishing of a message to your broker by subscribing to the “test_topic” and monitoring when the following is issued:

mosquitto_pub -h <<<broker-ip>>> -m some_test_text -t test_topic

Assuming you see “some_test_text” when you’re subscribed to “test_topic” then your tools are ready to use

Next, configure Nagios to publish a message instead of sending an email. There are a few options here – you can REPLACE the email action with an MQTT publish, or ADD the MQTT stuff as well as the email action. You can also simplify the setup by testing with a static message before amending the configuration again to use data from Nagios within the message.

The following enables a static, fairly content-free, message that will send instead of an email. I’d suggest using it to establish all is working before executing the more detailed steps for a fully-working, “useful” configuration further down.

Simply edit the commands.cfg file as described in the post about email notifications to replace the command_line entry for the notify-host-by-email command definition with the following, replacing <<<ipaddress>>> with the IP address of your MQTT broker

command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mosquitto_pub -h <<<ipaddress>>> -m "testing Nagios" -t "new_topic"

Then, again using instructions from the email notification post, test the custom send action and check your MQTT topic “new_topic”. You should receive an MQTT message that doesn’t say much but proves something is working!

Assuming you wish to go further and set up something useful, the following steps produced MQTT publishes with valid content for me, while keeping the emails coming too (note you will need to reverse the above steps and restore the commands.cfg lines to their original states first):

Firstly, add some user macros to the resources.cfg file similar to those used in the email notification setup. I added the following at the bottom of the existing content:

# added by me to enable MQTT using mosquitto_pub
$USER11$=<<<mqtt_broker_address>>>
$USER12$=<<<mqtt_topic_root>>>

Where mqtt_broker_address is the IP address (or hostname) of the system running MQTT, and mqtt_topic_root ends in a “/”. Then edit your commands.cfg file to append the following to the end of the command_line entry already there

 && /usr/bin/mosquitto_pub -h $USER11$ -m "$NOTIFICATIONTYPE$,$HOSTSTATE$,$HOSTOUTPUT$" -t "$USER12$$HOSTNAME$-host"

Don’t forget to restart the Nagios service and then force a custom notification to test it. Monitor all messages on your MQTT broker with the topic starting <<<mqtt_topic_root>>>, so if your mqtt_topic_root was “MySystem/Nagios/” then you’d subscribe to MySystem/Nagios/# to see them – the actual topic published to will then be MySystem/Nagios/hostname (where hostname is set by Nagios depending on which host you picked to send the custom notification from).

Once it’s working update the notify-service-by-email command_line in a similar way, with the following commands.cfg addition:

 && /usr/bin/mosquitto_pub -h $USER11$ -m "$NOTIFICATIONTYPE$,$SERVICEDESC$,$SERVICESTATE$" -t "$USER12$$HOSTNAME$-service/$SERVICEDESC$"

Finally, note…

There appears to be no way to trigger an email and MQTT publish with separate commands – while separate commands can be created, only one command can be triggered by a notification. So it’s one or the other or both – but not one or the other depending on different circumstances. This came about because I’d like to ALWAYS send an MQTT message when there’s a notification, but only send an email if there’s something critical – eg an important device or service down, or if there’s an unimportant device down for a long period.

However, by disabling email and only publishing MQTT I can make that decision elsewhere (processing the MQTT messages elsewhere in the system) and then sending emails only when certain criteria are met. While I haven’t done this yet the obvious place to do it is in the Node-RED environment I already have running and already have subscribing to my MQTT broker. More here when that’s working.