web stats
DIY Queue Depth Alerts - Mirth Community

Go Back   Mirth Community > Mirth Connect > Development

Reply
 
Thread Tools Display Modes
  #1  
Old 02-02-2011, 09:55 AM
TMarz TMarz is offline
OBX.2 Kenobi
 
Join Date: Jul 2009
Posts: 60
TMarz is on a distinguished road
Default DIY Queue Depth Alerts

I thought I'd share way I have put together for Mirth Alerts triggered by Queue Depth.

I would welcome any thoughts, comments or suggestions for improvements as it is far from perfect.

Setup:
I assumed the easiest way to accomplish this custom alert was with the postprocessor. My company has hundreds of instances of Mirth in the field with literally thousands of channels so in a plan to ease a future wide spread distribution, I put my code in the Global Postprocessor. It has some drawbacks that I will point out later.

One big concern I had was that I knew I didn't want a simple threshold checker. This would have caused me to get an alert for threshold, threshold+1, ....,threshold+n blowing up my mailbox until it was fixed. To solve this, I used modulo of the threshold. In the example code below I used 100, so that for each multiple of 100 an alert is generated. Not great, but better than just a >100 strategy.

Code:
// This script applies across all channels
var channelController = Packages.com.mirth.connect.server.controllers.ChannelController.getInstance();
var channelName = channelController.getDeployedChannelById(channelId).getName();
var channelNameUpper = channelName.toString().toUpperCase();

//Open database connection
var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
var expression = 'SELECT queued FROM channel_statistics WHERE channel_id = \''+channelId+'\';';
//channelResult is of java type ResultSet
var channelStatResult = dbConn.executeCachedQuery(expression);
var numOfQueued = 0;

while(channelStatResult.next())
{
	numOfQueued = channelStatResult.getInt("queued");
	if(numOfQueued % 100 == 0 && numOfQueued != 0)
	{
		//Generate an alert
		alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
	}
}

return;
We use postgres as our DB but I don't think this couldn't be adapted to any other. The key is that after each message, the post processor trawls the DB for the queued count. Admittedly, maybe not the most efficient process but I haven't noticed any performance issues. It might be possible, although I haven't explored it, to reference the message status. I know it can be done from the individual channel postprocessor as you have access to the messageObject variable as well as the responseMap which would tell you all you need to know. But I am not sure if you can do it from the Global Postproccessor. If you could, you might be able to improve efficiency by making the above code conditional on the message status being queued. Now that I think of it, you would probably have to set it up individually for each channel and its specific destinations because of the responseMap's getStatus arguments are literal strings. This wouldn't work for me because of how many Mirth servers and channels I have but might work for anyone who has only a handful of channels to maintain.

In addition to the above postproccessor, you will need to set up Alert's as you normally would. With the SMTP settings and it should be set to monitor any channels you wish. The alerts.sendAlert line will throw a Custom Error 302 and the string within that line can be accessed through the ${error} variable. I did it this way because I couldn't get Global Variables to work, but this did the trick. My Email Body looks something like:

PHP Code:
Alert!
-----------------------------------------
Alert generated at : ${date}

The following channel is experiencing a large volume of queued records to its destination(s).

${
error}
-----------------------------------------
**
This e-mail was automatically generated by the Alerting ServiceReplies to this message are not monitored.** 
And that is about all it takes. Now, there are some rough edges. First, if alot of messages queue up very quickly, or if there is simply alot of traffic, then I have noticed some multiples of 100 get skipped and no alert is ever sent. This probably has something to do with the fact that the Global PostProccessor is having to manage the 200 channels on my test server. If that is the case, moving it to the channel's own postprocessor should fix this.

Another issue I personally have, because of our setup, is that new channels can be added at any time to any of our instances of Mirth. I was wondering if anyone had any ideas about possible ways to utilize the global startscript to check for new channels and add them to the alert. I am currently thinking about trawling the DB for all the channel ID's then inserting into the requisite tables. I think there are three, alert, channel_alert, and alert_email. But I will need to check. Any thoughts or ideas would be welcomed.

Lastly, when I was working on this, I found something that troubled me. Because the queuestore is not a part of the DB, but rather a mule file system, it looks like that when a channel or destination is deleted, its queuestore remains and is not deleted. I am thinking about modifying my postgres-channel.xml file to clean this up but I was curious if this was a known issue, not considered an issue, of if this is not how it is supposed to work. I don't mind submitting the JIRA request myself if neccessary but maybe someone from the Mirth Team can chime in. Jacob, I'm looking at you


BTW, I am most certainly not the first to come up with something like this. In the interest of giving credit where credit is due, Chad recently came up with something similar here. I am sure there are others, but Chad's posting led me to submit what I had developed awhile back so that others could benefit and also maybe offer suggestions to ironing out the issues I mentioned.
Reply With Quote
  #2  
Old 02-02-2011, 02:11 PM
TMarz TMarz is offline
OBX.2 Kenobi
 
Join Date: Jul 2009
Posts: 60
TMarz is on a distinguished road
Default Auto-magically Add Newly Created Channels to an Alert

After I wrote the above post, I decided to pursue my idea of automatically adding new channels to this alert. I wrote the following script that is meant to run in the Global Deploy Script. The only requirement is that you must tell it what the name of your alert is.

After a deploy, the script gets the alert id for the name you have entered. It then queries the DB to get all channels associated with that alert id from the table channel_alert. It then gets all the channel ids. Finally it compares the two arrays of channel id and if it finds a channel on the channel list that is not in the alert list, it inserts it into the database.

I tested this and the Alert on Mirth 2.0.1 running Linux, but that shouldn't matter.

Code:
// This script executes once when the mule engine is started
// You only have access to the globalMap here to persist data

/***************************************\
*	This adds any new channels to the	                *
*	queue depth alert.					*
****************************************/

//Get the ID of the alert
//Open database connection
var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
var expression = 'SELECT id FROM alert WHERE name = \'Queue Depth Alert\';';
//channelResult is of java type ResultSet
var alertId = dbConn.executeCachedQuery(expression);
while(alertId.next())
{
	var id = alertId.getString("id");
}

//Get the list of channels
expression = 'SELECT id FROM channel ORDER BY id ASC;';
var channels = dbConn.executeCachedQuery(expression);
//Build an Array of channels
var channelArray = new Array();
while(channels.next())
{
	channelArray.push(channels.getString("id"));
}

//Get the list of channels for the Alert
expression = 'SELECT channel_id FROM channel_alert WHERE alert_id = \''+id+'\' ORDER BY channel_id ASC;';
var alertChannels = dbConn.executeCachedQuery(expression);
//Build an Array of channels
var channelAlertArray = new Array();
while(alertChannels.next())
{
	channelAlertArray.push(alertChannels.getString("channel_id"));
}

//Simple Exit Test
if(channelAlertArray.length != channelArray.length)
{
	 //index for the channelArray
	for(var i = 0; i < channelArray.length; i++)
	{
		var index = -1;
		//index for the Alert Channel Array
		for(var k = 0; k < channelAlertArray.length; k++)
		{
			if(channelAlertArray[k].toString() == channelArray[i].toString())
				index = k;
		}
		if( index == -1 )
		{
			//Insert into the table
			expression = 'INSERT INTO channel_alert VALUES (\''+channelArray[i]+'\', \''+id+'\');';
			dbConn.executeUpdate(expression);
		}
	}  
}
dbConn.close();
return;
Reply With Quote
  #3  
Old 02-16-2011, 12:08 PM
laidback_01 laidback_01 is offline
OBX.2 Kenobi
 
Join Date: Apr 2008
Location: Flathead Valley MT
Posts: 51
laidback_01
Default

This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.
Reply With Quote
  #4  
Old 02-17-2011, 12:55 PM
TMarz TMarz is offline
OBX.2 Kenobi
 
Join Date: Jul 2009
Posts: 60
TMarz is on a distinguished road
Default

Quote:
Originally Posted by laidback_01 View Post
This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.

I don't mind at all. Let me know if do have success making it work with Icinga. I'd be curious to see how.
Reply With Quote
  #5  
Old 02-20-2011, 06:04 AM
seaston seaston is offline
OBX.3 Kenobi
 
Join Date: Feb 2010
Location: London, UK
Posts: 168
seaston is on a distinguished road
Default

Thanks for the example. I will try using this, but have simplified by removing the database lookup for the queue count as shown by using the channelStatisticsController:

var channelStatisticsController = Packages.com.mirth.connect.server.controllers.Chan nelStatisticsController.getInstance();
var stats = channelStatisticsController.getStatistics(channelI d);
var numOfQueued = stats.queued;

if(numOfQueued % 100 == 0 && numOfQueued != 0)
{
//Generate an alert
alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
}

Simon

Last edited by seaston; 02-20-2011 at 06:06 AM.
Reply With Quote
  #6  
Old 11-14-2011, 02:44 PM
smeyer smeyer is offline
Mirth Newb
 
Join Date: Jan 2009
Posts: 12
smeyer
Default Python

In my experience the channel statistics queue count hasn't been quite what i'm interested in knowing (assuming that's the same value that shows up under the 'Queued' column on the dashboard). Instead when we get a lot of messages in to a channel the only way to see how many messages are backed up has been to count the files in the queuestore folder.

So here is what we're currently doing to track our queues. It uses python, mutt, and runs on linux.

Code is in python
Code:
import os
import commands
basedir = '/path-to-appdata/queuestore/'
limit = 30
queueToChannel = {'CHANNEL ID':'CHANNEL NAME'}
#loop through queue folders
for dirname in os.listdir(basedir):
        # ls -l    lists 1 file per line,  wc -l   counts the number of lines returned
        cmd = 'ls -l ' + basedir + dirname + ' | wc -l '
        #get results of wc -l  
        filecount = commands.getoutput(cmd)
        try:
                filecount = int(filecount) -1;
        except:
                filecount = 0;
        if (filecount > limit):
                commands.getoutput('echo "mirth queuestore ' + dirname + ' representing ' + queueToChannel[dirname] + ' has ' + str(filecount) + ' files" | mutt -s "mirth queuestore for ' + queueToChannel[dirname] + ' is above limit of ' + str(limit) +'" your@emailname.com')
Anywho, this has been helpful for us but does require manual entry of the queueToChannel mappings. Not sure if it'll be helpful for you guys or not, but better to post and find out it wasn't needed than vice versa

-Scott
Reply With Quote
Reply

Tags
alert, alerts, queue, queued, queuedepth

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 03:24 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation