Xymon is a brilliantly simple network management tool that runs reliably in the background and can graph, alert and track pretty much anything you want. I use Xymon for example to monitor a number of things around my home - including all my servers and devices, the quality of my Internet - and the temperature in and around the house. I've developed many of the scripts I use to carry out this monitoring myself using Perl. However, while Xymon supports the monitoring of remote devices via ICMP or TCP port tests or custom scripts, it doesn't currently provide any sensible form of SNMP monitoring. Luckily, help is at hand in the shape of Devmon. Devmon is a Perl daemon that is designed to supplement and enhance the capabilities of either a BigBrother or Hobbit/Xymon monitoring server allowing that server to monitor remote devices via SNMP (Simple Network Management Protocol) and integrate this information within the same displays on the BigBrother/Hobbit display server.
However, the documentation with Devmon is sparse and rather lacking, especially around getting it started when the server starts up. There are example scripts for Red Hat available in the package but nothing for Ubuntu. I've set about rectifying this and I present my start-up approach to Devmon using Ubuntu's Upstart system.
While Ubuntu supports the more traditional init-based scripts to start and stop services, it is better to think about using Upstart instead, as it is a more robust services management daemon that allows for things like dependencies, custom events/triggers, pre & post initialisation steps and resource limitations, amongst other things.
Installation of Xymon and Devmon
The full installation of Xymon and Devmon is beyond the scope of this article, so I'll summarise what I've done here so you can adapt my scripts and notes for your own use:
- Xymon is installed under /home/xymon with a username of xymon. It stores its logfiles in /var/log/xymon/
- Devmon is installed under /usr/lib/devmon with a username of xymon (so editing scripts and templates is all completed under the one username) and logs to /var/log/devmon/devmon.log
There are plenty of Upstart tutorials on the Internet - and an excellent 'Cookbook' to check out, so I won't repeat that here. All that is necessary is to say that:
- Upstart's scripts are stored in the /etc/init directory and all end in .conf.
- Upstart logs its output to logs for each script in /var/log/upstart/
That's all it took. Looking closer at the script:#!upstart description "DEVMON Hobbit/Xymon SNMP tool upstart script" author "Martin Davies" start on runlevel [2345] stop on runlevel [!2345] env PIDFILE="/var/run/devmon/devmon.pid" env DEVMON_USER="xymon" env DEVMON_DIR="/usr/lib/devmon/" script if [ ! -d "${DEVMON_DIR}" ]; then echo "${DEVMON_DIR} missing, aborting." exit 1 fi exec start-stop-daemon --start -c ${DEVMON_USER} -d ${DEVMON_DIR} --exec ${DEVMON_DIR}devmon -- -f end script
These two lines comment the script and give enough info for anyone to see what the script is for
description "DEVMON Hobbit/Xymon SNMP tool upstart script" author "Martin Davies"
These next lines simply tell Upstart when to run the script (the figures in brackes are the runlevels of Ubuntu)
start on runlevel [2345] stop on runlevel [!2345]
Then we set up the environment variables to set the PID file, the user Devmon runs under and the directory that devmon is installed in
env PIDFILE="/var/run/devmon/devmon.pid" env DEVMON_USER="xymon" env DEVMON_DIR="/usr/lib/devmon/"
Finally the script itself. First we check that the install directory exists and if it does we run devmon as per the user and directory settings set above. While testing, I recommend that you add -vvv --debug to the end of 'exec start-stop-daemon' line.
script if [ ! -d "${DEVMON_DIR}" ]; then echo "${DEVMON_DIR} missing, aborting." exit 1 fi exec start-stop-daemon --start -c ${DEVMON_USER} -d ${DEVMON_DIR} --exec ${DEVMON_DIR}devmon -- -f end script
Addendum:
I had cause to shut down my server the other day to replace a disk and when it was restarted Devmon didn't start. The problem I saw was that the Devmon entries on Xymon pages stayed purple. Even when starting Devmon by hand I still had purple entries for devmon in the xymon web pages
martin@homeserver:~$ sudo start devmon devmon start/running, process 28703 martin@homeserver:~$You can see here that Devmon reports as starting but after a few minutes I still had purple icons. Looking at the Devmon log file gave me the answer:
It turned out that a kill script from the old init.d based setup had not only deleted the pidfile but had also deleted the directory too. So I removed the killscript and re-created the pidfile directory:martin@homeserver:~$ cat /var/log/devmon/devmon.log [14-07-28@21:51:39] Shutting down [14-07-30@22:18:12] Cant write to pidfile /var/run/devmon/devmon.pid (No such file or directory) [14-08-05@18:09:00] Cant write to pidfile /var/run/devmon/devmon.pid (No such file or directory)
martin@homeserver:~$ sudo mkdir /var/run/devmon/ martin@homeserver:~$ sudo start devmon devmon start/running, process 30369And looked at the log file again and saw this line
Meh! OK - one more step and we're done:[14-08-06@21:40:16] Cant write to pidfile /var/run/devmon/devmon.pid (Permission denied)
martin@homeserver:~$ sudo chown xymon:xymon /var/run/devmon martin@homeserver:~$ sudo start devmon devmon start/running, process 32309 martin@homeserver:~$ cat /var/log/devmon/devmon.logFinally. The last few lines on the log file read
And sure enough, after a few minutes, the Devmon icons came back to life. I'll check the script again and see if I need to put further tests in the script. If I do, I'll update the post and let you know.[14-08-06@21:40:16] Cant write to pidfile /var/run/devmon/devmon.pid (Permission denied) [14-08-06@21:47:09] ---Initilizing devmon... [14-08-06@21:47:09] Node 0 reporting to localhost [14-08-06@21:47:09] Running under process id: 32309 [14-08-06@21:47:09] Entering poll loop
As usual, please get in touch if this has been useful and drop me a comment if you need more info - or if you find a mistake!
