Mirth Community

Mirth Community (http://www.mirthcorp.com/community/forums/index.php)
-   Development (http://www.mirthcorp.com/community/forums/forumdisplay.php?f=8)
-   -   DIY Queue Depth Alerts (http://www.mirthcorp.com/community/forums/showthread.php?t=5037)

TMarz 02-02-2011 09:55 AM

DIY Queue Depth Alerts
 
I thought I'd share way I have put together for Mirth Alerts triggered by Queue Depth.

I would welcome any thoughts, comments or suggestions for improvements as it is far from perfect.

Setup:
I assumed the easiest way to accomplish this custom alert was with the postprocessor. My company has hundreds of instances of Mirth in the field with literally thousands of channels so in a plan to ease a future wide spread distribution, I put my code in the Global Postprocessor. It has some drawbacks that I will point out later.

One big concern I had was that I knew I didn't want a simple threshold checker. This would have caused me to get an alert for threshold, threshold+1, ....,threshold+n blowing up my mailbox until it was fixed. To solve this, I used modulo of the threshold. In the example code below I used 100, so that for each multiple of 100 an alert is generated. Not great, but better than just a >100 strategy.

Code:

// This script applies across all channels
var channelController = Packages.com.mirth.connect.server.controllers.ChannelController.getInstance();
var channelName = channelController.getDeployedChannelById(channelId).getName();
var channelNameUpper = channelName.toString().toUpperCase();

//Open database connection
var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
var expression = 'SELECT queued FROM channel_statistics WHERE channel_id = \''+channelId+'\';';
//channelResult is of java type ResultSet
var channelStatResult = dbConn.executeCachedQuery(expression);
var numOfQueued = 0;

while(channelStatResult.next())
{
        numOfQueued = channelStatResult.getInt("queued");
        if(numOfQueued % 100 == 0 && numOfQueued != 0)
        {
                //Generate an alert
                alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
        }
}

return;

We use postgres as our DB but I don't think this couldn't be adapted to any other. The key is that after each message, the post processor trawls the DB for the queued count. Admittedly, maybe not the most efficient process but I haven't noticed any performance issues. It might be possible, although I haven't explored it, to reference the message status. I know it can be done from the individual channel postprocessor as you have access to the messageObject variable as well as the responseMap which would tell you all you need to know. But I am not sure if you can do it from the Global Postproccessor. If you could, you might be able to improve efficiency by making the above code conditional on the message status being queued. Now that I think of it, you would probably have to set it up individually for each channel and its specific destinations because of the responseMap's getStatus arguments are literal strings. This wouldn't work for me because of how many Mirth servers and channels I have but might work for anyone who has only a handful of channels to maintain.

In addition to the above postproccessor, you will need to set up Alert's as you normally would. With the SMTP settings and it should be set to monitor any channels you wish. The alerts.sendAlert line will throw a Custom Error 302 and the string within that line can be accessed through the ${error} variable. I did it this way because I couldn't get Global Variables to work, but this did the trick. My Email Body looks something like:

PHP Code:

Alert!
-----------------------------------------
Alert generated at : ${date}

The following channel is experiencing a large volume of queued records to its destination(s).

${
error}
-----------------------------------------
**
This e-mail was automatically generated by the Alerting ServiceReplies to this message are not monitored.** 

And that is about all it takes. Now, there are some rough edges. First, if alot of messages queue up very quickly, or if there is simply alot of traffic, then I have noticed some multiples of 100 get skipped and no alert is ever sent. This probably has something to do with the fact that the Global PostProccessor is having to manage the 200 channels on my test server. If that is the case, moving it to the channel's own postprocessor should fix this.

Another issue I personally have, because of our setup, is that new channels can be added at any time to any of our instances of Mirth. I was wondering if anyone had any ideas about possible ways to utilize the global startscript to check for new channels and add them to the alert. I am currently thinking about trawling the DB for all the channel ID's then inserting into the requisite tables. I think there are three, alert, channel_alert, and alert_email. But I will need to check. Any thoughts or ideas would be welcomed.

Lastly, when I was working on this, I found something that troubled me. Because the queuestore is not a part of the DB, but rather a mule file system, it looks like that when a channel or destination is deleted, its queuestore remains and is not deleted. I am thinking about modifying my postgres-channel.xml file to clean this up but I was curious if this was a known issue, not considered an issue, of if this is not how it is supposed to work. I don't mind submitting the JIRA request myself if neccessary but maybe someone from the Mirth Team can chime in. Jacob, I'm looking at you:D


BTW, I am most certainly not the first to come up with something like this. In the interest of giving credit where credit is due, Chad recently came up with something similar here. I am sure there are others, but Chad's posting led me to submit what I had developed awhile back so that others could benefit and also maybe offer suggestions to ironing out the issues I mentioned.

TMarz 02-02-2011 02:11 PM

Auto-magically Add Newly Created Channels to an Alert
 
After I wrote the above post, I decided to pursue my idea of automatically adding new channels to this alert. I wrote the following script that is meant to run in the Global Deploy Script. The only requirement is that you must tell it what the name of your alert is.

After a deploy, the script gets the alert id for the name you have entered. It then queries the DB to get all channels associated with that alert id from the table channel_alert. It then gets all the channel ids. Finally it compares the two arrays of channel id and if it finds a channel on the channel list that is not in the alert list, it inserts it into the database.

I tested this and the Alert on Mirth 2.0.1 running Linux, but that shouldn't matter.

Code:

// This script executes once when the mule engine is started
// You only have access to the globalMap here to persist data

/***************************************\
*        This adds any new channels to the                        *
*        queue depth alert.                                        *
****************************************/

//Get the ID of the alert
//Open database connection

var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
var expression = 'SELECT id FROM alert WHERE name = \'Queue Depth Alert\';';
//channelResult is of java type ResultSet
var alertId = dbConn.executeCachedQuery(expression);
while(alertId.next())
{
        var id = alertId.getString("id");
}

//Get the list of channels
expression = 'SELECT id FROM channel ORDER BY id ASC;';
var channels = dbConn.executeCachedQuery(expression);
//Build an Array of channels
var channelArray = new Array();
while(channels.next())
{
        channelArray.push(channels.getString("id"));
}

//Get the list of channels for the Alert
expression = 'SELECT channel_id FROM channel_alert WHERE alert_id = \''+id+'\' ORDER BY channel_id ASC;';
var alertChannels = dbConn.executeCachedQuery(expression);
//Build an Array of channels
var channelAlertArray = new Array();
while(alertChannels.next())
{
        channelAlertArray.push(alertChannels.getString("channel_id"));
}

//Simple Exit Test
if(channelAlertArray.length != channelArray.length)
{
        //index for the channelArray
        for(var i = 0; i < channelArray.length; i++)
        {
                var index = -1;
                //index for the Alert Channel Array
                for(var k = 0; k < channelAlertArray.length; k++)
                {
                        if(channelAlertArray[k].toString() == channelArray[i].toString())
                                index = k;
                }
                if( index == -1 )
                {
                        //Insert into the table
                        expression = 'INSERT INTO channel_alert VALUES (\''+channelArray[i]+'\', \''+id+'\');';
                        dbConn.executeUpdate(expression);
                }
        } 
}
dbConn.close();
return;


laidback_01 02-16-2011 12:08 PM

This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.

TMarz 02-17-2011 12:55 PM

Quote:

Originally Posted by laidback_01 (Post 17814)
This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.


I don't mind at all. Let me know if do have success making it work with Icinga. I'd be curious to see how.

seaston 02-20-2011 06:04 AM

Thanks for the example. I will try using this, but have simplified by removing the database lookup for the queue count as shown by using the channelStatisticsController:

var channelStatisticsController = Packages.com.mirth.connect.server.controllers.Chan nelStatisticsController.getInstance();
var stats = channelStatisticsController.getStatistics(channelI d);
var numOfQueued = stats.queued;

if(numOfQueued % 100 == 0 && numOfQueued != 0)
{
//Generate an alert
alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
}

Simon

smeyer 11-14-2011 02:44 PM

Python
 
In my experience the channel statistics queue count hasn't been quite what i'm interested in knowing (assuming that's the same value that shows up under the 'Queued' column on the dashboard). Instead when we get a lot of messages in to a channel the only way to see how many messages are backed up has been to count the files in the queuestore folder.

So here is what we're currently doing to track our queues. It uses python, mutt, and runs on linux.

Code is in python
Code:

import os
import commands
basedir = '/path-to-appdata/queuestore/'
limit = 30
queueToChannel = {'CHANNEL ID':'CHANNEL NAME'}
#loop through queue folders
for dirname in os.listdir(basedir):
        # ls -l    lists 1 file per line,  wc -l  counts the number of lines returned
        cmd = 'ls -l ' + basedir + dirname + ' | wc -l '
        #get results of wc -l 
        filecount = commands.getoutput(cmd)
        try:
                filecount = int(filecount) -1;
        except:
                filecount = 0;
        if (filecount > limit):
                commands.getoutput('echo "mirth queuestore ' + dirname + ' representing ' + queueToChannel[dirname] + ' has ' + str(filecount) + ' files" | mutt -s "mirth queuestore for ' + queueToChannel[dirname] + ' is above limit of ' + str(limit) +'" your@emailname.com')

Anywho, this has been helpful for us but does require manual entry of the queueToChannel mappings. Not sure if it'll be helpful for you guys or not, but better to post and find out it wasn't needed than vice versa :)

-Scott


All times are GMT -8. The time now is 10:03 AM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation