Getting Devmon to start automatically with Xymon on Ubuntu

Introduction
Xymon is a brilliantly simple network management tool that runs reliably in the background and can graph, alert and track pretty much anything you want. I use Xymon for example to monitor a number of things around my home - including all my servers and devices, the quality of my Internet - and the temperature in and around the house.  I've developed many of the scripts I use to carry out this monitoring myself using Perl.  However, while Xymon supports the monitoring of remote devices via ICMP or TCP port tests or custom scripts, it doesn't currently provide any sensible form of SNMP monitoring.  Luckily, help is at hand in the shape of Devmon. Devmon is a Perl daemon that is designed to supplement and enhance the capabilities of either a BigBrother or Hobbit/Xymon monitoring server allowing that server to monitor remote devices via SNMP (Simple Network Management Protocol) and integrate this information within the same displays on the BigBrother/Hobbit display server.

However, the documentation with Devmon is sparse and rather lacking, especially around getting it started when the server starts up.  There are example scripts for Red Hat available in the package but nothing for Ubuntu.  I've set about rectifying this and I present my start-up approach to Devmon using Ubuntu's Upstart system.

While Ubuntu supports the more traditional init-based scripts to start and stop services, it is better to think about using Upstart instead, as it is a more robust services management daemon that allows for things like dependencies, custom events/triggers, pre & post initialisation steps  and resource limitations, amongst other things.

Installation of  Xymon and Devmon
The full installation of Xymon and Devmon is beyond the scope of this article, so I'll summarise what I've done here so you can adapt my scripts and notes for your own use:

• Xymon is installed under /home/xymon with a username of xymon.  It stores its logfiles in /var/log/xymon/
• Devmon is installed under /usr/lib/devmon with a username of xymon (so editing scripts and templates is all completed under the one username) and logs to /var/log/devmon/devmon.log

Configuring Upstart
There are plenty of Upstart tutorials on the Internet - and an excellent 'Cookbook' to check out, so I won't repeat that here.  All that is necessary is to say that:

• Upstart's scripts are stored in the /etc/init directory and all end in .conf.
• Upstart logs its output to logs for each script in /var/log/upstart/

To work out what I needed to get devmon running I set up PuTTY to have three sessions open to my server.  One I used to edit and test the script, one I had set to display the devmon logfile (tail -f /var/log/devmon/devmon.log) and one was set to show the upstart script (tail -f /var/log/upstart/devmon.log).  I found the easiest way to get the script I needed was to take another script as a template and then edit it to start devmon instead.  My script, devmon.conf, is shown below.

#!upstart
description "DEVMON Hobbit/Xymon SNMP tool upstart script"
author "Martin Davies"

start on runlevel [2345]
stop on runlevel [!2345]

env PIDFILE="/var/run/devmon/devmon.pid"
env DEVMON_USER="xymon"
env DEVMON_DIR="/usr/lib/devmon/"

script
        if [ ! -d "${DEVMON_DIR}" ]; then
                echo "${DEVMON_DIR} missing, aborting."
                exit 1
        fi

        exec start-stop-daemon --start -c ${DEVMON_USER} -d ${DEVMON_DIR} --exec ${DEVMON_DIR}devmon -- -f

end script

That's all it took.  Looking closer at the script:

These two lines comment the script and give enough info for anyone to see what the script is for

description "DEVMON Hobbit/Xymon SNMP tool upstart script"
author "Martin Davies"

These next lines simply tell Upstart when to run the script (the figures in brackes are the runlevels of Ubuntu)

start on runlevel [2345]
stop on runlevel [!2345]

Then we set up the environment variables to set the PID file, the user Devmon runs under and the directory that devmon is installed in

env PIDFILE="/var/run/devmon/devmon.pid"
env DEVMON_USER="xymon"
env DEVMON_DIR="/usr/lib/devmon/"

Finally the script itself.  First we check that the install directory exists and if it does we run devmon as per the user and directory settings set above.  While testing, I recommend that you add -vvv --debug to the end of 'exec start-stop-daemon' line.

script
    if [ ! -d "${DEVMON_DIR}" ]; then
        echo "${DEVMON_DIR} missing, aborting."
        exit 1
    fi
    exec start-stop-daemon --start -c ${DEVMON_USER} -d ${DEVMON_DIR} --exec ${DEVMON_DIR}devmon -- -f
end script

Addendum:
I had cause to shut down my server the other day to replace a disk and when it was restarted Devmon didn't start. The problem I saw was that the Devmon entries on Xymon pages stayed purple. Even when starting Devmon by hand I still had purple entries for devmon in the xymon web pages

martin@homeserver:~$ sudo start devmon
devmon start/running, process 28703
martin@homeserver:~$ 

You can see here that Devmon reports as starting but after a few minutes I still had purple icons. Looking at the Devmon log file gave me the answer:

martin@homeserver:~$ cat /var/log/devmon/devmon.log
[14-07-28@21:51:39] Shutting down
[14-07-30@22:18:12] Cant write to pidfile /var/run/devmon/devmon.pid (No such file or directory)
[14-08-05@18:09:00] Cant write to pidfile /var/run/devmon/devmon.pid (No such file or directory)

It turned out that a kill script from the old init.d based setup had not only deleted the pidfile but had also deleted the directory too. So I removed the killscript and re-created the pidfile directory:

martin@homeserver:~$ sudo mkdir /var/run/devmon/
martin@homeserver:~$ sudo start devmon
devmon start/running, process 30369

And looked at the log file again and saw this line

[14-08-06@21:40:16] Cant write to pidfile /var/run/devmon/devmon.pid (Permission denied)

Meh! OK - one more step and we're done:

martin@homeserver:~$ sudo chown xymon:xymon /var/run/devmon
martin@homeserver:~$ sudo start devmon
devmon start/running, process 32309
martin@homeserver:~$ cat /var/log/devmon/devmon.log

Finally. The last few lines on the log file read

[14-08-06@21:40:16] Cant write to pidfile /var/run/devmon/devmon.pid (Permission denied)
[14-08-06@21:47:09] ---Initilizing devmon...
[14-08-06@21:47:09] Node 0 reporting to localhost
[14-08-06@21:47:09] Running under process id: 32309
[14-08-06@21:47:09] Entering poll loop

And sure enough, after a few minutes, the Devmon icons came back to life. I'll check the script again and see if I need to put further tests in the script.  If I do, I'll update the post and let you know.

As usual, please get in touch if this has been useful and drop me a comment if you need more info - or if you find a mistake!