web stats
Mirth Community - View Single Post - DIY Queue Depth Alerts
View Single Post
Old 02-02-2011, 08:55 AM
TMarz TMarz is offline
OBX.2 Kenobi
Join Date: Jul 2009
Posts: 60
TMarz is on a distinguished road
Default DIY Queue Depth Alerts

I thought I'd share way I have put together for Mirth Alerts triggered by Queue Depth.

I would welcome any thoughts, comments or suggestions for improvements as it is far from perfect.

I assumed the easiest way to accomplish this custom alert was with the postprocessor. My company has hundreds of instances of Mirth in the field with literally thousands of channels so in a plan to ease a future wide spread distribution, I put my code in the Global Postprocessor. It has some drawbacks that I will point out later.

One big concern I had was that I knew I didn't want a simple threshold checker. This would have caused me to get an alert for threshold, threshold+1, ....,threshold+n blowing up my mailbox until it was fixed. To solve this, I used modulo of the threshold. In the example code below I used 100, so that for each multiple of 100 an alert is generated. Not great, but better than just a >100 strategy.

// This script applies across all channels
var channelController = Packages.com.mirth.connect.server.controllers.ChannelController.getInstance();
var channelName = channelController.getDeployedChannelById(channelId).getName();
var channelNameUpper = channelName.toString().toUpperCase();

//Open database connection
var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
var expression = 'SELECT queued FROM channel_statistics WHERE channel_id = \''+channelId+'\';';
//channelResult is of java type ResultSet
var channelStatResult = dbConn.executeCachedQuery(expression);
var numOfQueued = 0;

	numOfQueued = channelStatResult.getInt("queued");
	if(numOfQueued % 100 == 0 && numOfQueued != 0)
		//Generate an alert
		alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);

We use postgres as our DB but I don't think this couldn't be adapted to any other. The key is that after each message, the post processor trawls the DB for the queued count. Admittedly, maybe not the most efficient process but I haven't noticed any performance issues. It might be possible, although I haven't explored it, to reference the message status. I know it can be done from the individual channel postprocessor as you have access to the messageObject variable as well as the responseMap which would tell you all you need to know. But I am not sure if you can do it from the Global Postproccessor. If you could, you might be able to improve efficiency by making the above code conditional on the message status being queued. Now that I think of it, you would probably have to set it up individually for each channel and its specific destinations because of the responseMap's getStatus arguments are literal strings. This wouldn't work for me because of how many Mirth servers and channels I have but might work for anyone who has only a handful of channels to maintain.

In addition to the above postproccessor, you will need to set up Alert's as you normally would. With the SMTP settings and it should be set to monitor any channels you wish. The alerts.sendAlert line will throw a Custom Error 302 and the string within that line can be accessed through the ${error} variable. I did it this way because I couldn't get Global Variables to work, but this did the trick. My Email Body looks something like:

PHP Code:
Alert generated at : ${date}

The following channel is experiencing a large volume of queued records to its destination(s).

This e-mail was automatically generated by the Alerting ServiceReplies to this message are not monitored.** 
And that is about all it takes. Now, there are some rough edges. First, if alot of messages queue up very quickly, or if there is simply alot of traffic, then I have noticed some multiples of 100 get skipped and no alert is ever sent. This probably has something to do with the fact that the Global PostProccessor is having to manage the 200 channels on my test server. If that is the case, moving it to the channel's own postprocessor should fix this.

Another issue I personally have, because of our setup, is that new channels can be added at any time to any of our instances of Mirth. I was wondering if anyone had any ideas about possible ways to utilize the global startscript to check for new channels and add them to the alert. I am currently thinking about trawling the DB for all the channel ID's then inserting into the requisite tables. I think there are three, alert, channel_alert, and alert_email. But I will need to check. Any thoughts or ideas would be welcomed.

Lastly, when I was working on this, I found something that troubled me. Because the queuestore is not a part of the DB, but rather a mule file system, it looks like that when a channel or destination is deleted, its queuestore remains and is not deleted. I am thinking about modifying my postgres-channel.xml file to clean this up but I was curious if this was a known issue, not considered an issue, of if this is not how it is supposed to work. I don't mind submitting the JIRA request myself if neccessary but maybe someone from the Mirth Team can chime in. Jacob, I'm looking at you

BTW, I am most certainly not the first to come up with something like this. In the interest of giving credit where credit is due, Chad recently came up with something similar here. I am sure there are others, but Chad's posting led me to submit what I had developed awhile back so that others could benefit and also maybe offer suggestions to ironing out the issues I mentioned.
Reply With Quote