web stats
Mirth message volume - Mirth Community

Go Back   Mirth Community > Mirth Connect > General Discussion

Reply
 
Thread Tools Display Modes
  #1  
Old 05-08-2008, 08:21 AM
MichaelP MichaelP is offline
What's HL7?
 
Join Date: May 2008
Posts: 5
MichaelP
Default Mirth message volume

I am investigating using Mirth to replace an existing HL7 interface. The current interface has hit the proverbial wall with message volume...to the point where it's throwing errors, failing to process some messages, and even completely dropping some messages.

The current interface receives hospital ADT data (roughly 4000-6000 message per day), and maintains multiple databases (SQL Server) for various department functions as well as interactive reporting services. It's very database intensive. Virtually every ADT message that comes over prompts some type of datawrite activity.

Should I expect Mirth to handle this volume?
Reply With Quote
  #2  
Old 05-08-2008, 08:59 AM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 719
jbartels is on a distinguished road
Default Re:Mirth message volume

4000-6000 per day? Thats it? :P

We've got a Mirth instance running on light hardware (VM with Ubnutu 7.10, 512MB RAM and 10GB disk) that handled 30000 messages per day during an import operation without blinking. Those operations were largely simple routing (message over VPN, into our datacentre, routed to correct customer EMR instance) and basic database writes.

What is your current interface doing? Is it simply stuffing messages into a database or is it applying some conversion/transformation/logic to those messages? I assume your reporting is being done with a reporting tool and not your existing interface?

You say that the existing setup is "database intensive". Does that simply mean that most of the HL7 data gets stuffed into a database or that the interface itself is required to do a bunch of work with the database before writing the data?

In the last 3 years we've only come across one case where we decided that it was better to roll our own interface instead of using Mirth and that was largely because of a janky deployment scenario than technical limitations.

In addition to being technically superior, Mirth comes with a reasonably active community (as you can see) and the Mirth Team/Webreach is very responsive in fixing bugs and offering support. So if you're able to provide more details about what your interface needs to do, there are likely people around who have done something similar (or at least one portion of it) with Mirth already.

I keep telling people this in my company, at my customers, and the people I know in the software community. Mirth kicks ass.
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #3  
Old 05-08-2008, 10:55 AM
MichaelP MichaelP is offline
What's HL7?
 
Join Date: May 2008
Posts: 5
MichaelP
Default Re:Mirth message volume

Quote:
Mirth kicks ass
So far, I'm inclined to agree with you on that...

The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.

That database is the backend for several in-house systems that support multiple department's data needs (Bed Control, Env Service, Case Management, Quality Management, etc..) and also serves as the data warehouse for multiple reporting tools.

Our current interface has hit the wall in terms of message volume processing. It randomly throws errors when it tries to open a connection to the database that hasn't finished closing from the prior message execution...and it randomly drops messages completely (presumably becuase it's tied up processing something).

Both issues are related to volume. Until the interface was recently expanded to cover outpatient messages, it clicked along like clockwork for several years with virtually no issues. If I modifying it to reduce the number of messages it needs to process, the errors go away.

We've explored upgrading the machine the interface runs on, but it's not really a memmory issue, and the software isn't able to take advantage of multiple processors. That's primarily why we're looking into alternative interface solutions (not to mention that problem of the current interface being a huge memory leak...nessecitating the server be rebooted every 12 hours).

Probably more than you wanted to know...but that's how we arrived at testing Mirth.

I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model
Reply With Quote
  #4  
Old 05-08-2008, 11:57 AM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 719
jbartels is on a distinguished road
Default Re:Mirth message volume

MichaelP wrote:
The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.
[/quote]

So if I'm reading that right, all it does is run a few queries or stored procedures. Any heavy lifting is done by that query/stored proc on the database server and no heavy logic is done at your interface.

Quote:
Probably more than you wanted to know...but that's how we arrived at testing Mirth.
Thanks for the review. My team writes software that has to interoperate with things like Mirth, PMs, EMRs, and HISs so the more I hear about issues the more I can do to make our code not suck to make everyones life a lot easier.

Quote:
I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model
If you look at my posts from about 2 years ago I developed a plan to load test Mirth. It never got implemented. The easier way to test it is to siphon off messages from your existing interface (either with Mirth as a simple router or some TCP load-balancer hacking), have Mirth run those messages into a testing database, and then compare the testing database with the production one.

Another option that will leverage Mirth and allow you to gradually phase it in would be to use it as a throttle on your traffic. If you can spread those 6000 messages out from 8 hours to 24 hours your existing interface gets to live a little longer and Mirth gets a shakedown. It'd take a bit of thought to get it right, but you can simply set up a wait-loop in a Mirth channel using javascript.

As far as deployment, multi-processor, etc. Mirth runs on Java. As long as you can deploy Java apps on a server with a good JVM (Sun-Java only IMHO) Mirth will run quite well. Like all Java apps it WILL suck up all your free RAM if you let it, set the XMax option appropriately. You'll also want to run Mirth itself on something other than its internal Derby database. I've had good results with both Postgres and SQL Server.
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #5  
Old 05-08-2008, 12:10 PM
jacobb jacobb is offline
Mirth Employee
 
Join Date: Aug 2006
Location: Irvine, CA
Posts: 1,218
jacobb is an unknown quantity at this point
Default Re:Mirth message volume

MichaelP wrote:
Quote:
Quote:
Mirth kicks ass
So far, I'm inclined to agree with you on that...

The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.

That database is the backend for several in-house systems that support multiple department's data needs (Bed Control, Env Service, Case Management, Quality Management, etc..) and also serves as the data warehouse for multiple reporting tools.

Our current interface has hit the wall in terms of message volume processing. It randomly throws errors when it tries to open a connection to the database that hasn't finished closing from the prior message execution...and it randomly drops messages completely (presumably becuase it's tied up processing something).

Both issues are related to volume. Until the interface was recently expanded to cover outpatient messages, it clicked along like clockwork for several years with virtually no issues. If I modifying it to reduce the number of messages it needs to process, the errors go away.

We've explored upgrading the machine the interface runs on, but it's not really a memmory issue, and the software isn't able to take advantage of multiple processors. That's primarily why we're looking into alternative interface solutions (not to mention that problem of the current interface being a huge memory leak...nessecitating the server be rebooted every 12 hours).

Probably more than you wanted to know...but that's how we arrived at testing Mirth.

I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model
First of all, you're going to want to make sure to use postgresql for the backend of mirth.

Next, if you are getting any problems connecting to the database, then there might be a problem establishing multiple connections to the database. Perhaps there are some DB settings you could mess with. Another thing you might want to look at for increasing performance is implementing a form of database connection pooling. You can create a database connection and then store it in the global map on deploy. To execute a javascript database update/query, simply grab the connection out of the globalmap, and call your execute method on it. You may also want to make sure it's still connected each time, and if not, reconnect. This alone may really help performance if you have any issues.

Good luck!

Post edited by: jacobb, at: 05/08/2008 13:16
__________________
Jacob Brauer
Director, Software Development
NextGen Healthcare

Reply With Quote
  #6  
Old 05-08-2008, 12:13 PM
jacobb jacobb is offline
Mirth Employee
 
Join Date: Aug 2006
Location: Irvine, CA
Posts: 1,218
jacobb is an unknown quantity at this point
Default Re:Mirth message volume

And on a side note, we've gotten huge performance increases running Mirth on multiple processors or multiple core machines. Our performance tests on systems with 8 cores have had considerable increases over systems with 4 cores.
__________________
Jacob Brauer
Director, Software Development
NextGen Healthcare

Reply With Quote
  #7  
Old 05-09-2008, 03:54 AM
MichaelP MichaelP is offline
What's HL7?
 
Join Date: May 2008
Posts: 5
MichaelP
Default Re:Mirth message volume

Quote:
So if I'm reading that right, all it does is run a few queries or stored procedures. Any heavy lifting is done by that query/stored proc on the database server and no heavy logic is done at your interface.
Sadly, the existing (legacy) system does a fair amount of the heavy lifting itself. Over the years I've supported it I've incrementally moved things over to the database side. A complete re-write would do it good, but there are other issues with the software and I'd just as soon move to something better anyway.

Quote:
If you look at my posts from about 2 years ago I developed a plan to load test Mirth. It never got implemented. The easier way to test it is to siphon off messages from your existing interface (either with Mirth as a simple router or some TCP load-balancer ), have Mirth run those messages into a testing database, and then compare the testing database with the production one.
I'm mirroring over my production database structure onto a test server now. Once I fully configure Mirth's Channel to accept the data I can configure a duplicate outbound interface from our HIS and give Mirth a full load test.

Quote:
You'll also want to run Mirth itself on something other than its internal Derby database. I've had good results with both Postgres and SQL Server.
That's good to know.

Quote:
And on a side note, we've gotten huge performance increases running Mirth on multiple processors or multiple core machines. Our performance tests on systems with 8 cores have had considerable increases over systems with 4 cores.
That's exactly what I wanted to know. I'm budgetting for a new interface machine this summer.
Reply With Quote
  #8  
Old 05-09-2008, 04:30 AM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 719
jbartels is on a distinguished road
Default Re:Mirth message volume

MichaelP wrote:
Quote:
That's exactly what I wanted to know. I'm budgetting for a new interface machine this summer.
http://www.webreachinc.com/products/mirthappliances
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #9  
Old 05-09-2008, 10:19 AM
jacobb jacobb is offline
Mirth Employee
 
Join Date: Aug 2006
Location: Irvine, CA
Posts: 1,218
jacobb is an unknown quantity at this point
Default Re:Mirth message volume

MichaelP, I too would recommend looking into an Enterprise cluster from us. They are all optimized for Mirth, so you'll get great message throughput and a really stable machine with a lot of neat extra features.
__________________
Jacob Brauer
Director, Software Development
NextGen Healthcare

Reply With Quote
  #10  
Old 05-14-2008, 06:18 AM
quimicefa quimicefa is offline
Mirth Guru
 
Join Date: Dec 2007
Location: Barcelona
Posts: 235
quimicefa is on a distinguished road
Default Re:Mirth message volume

In addition to the above, I've got a rate of +300.000 msg/hour taking data with a SOAP listener and writing the message to a remote SFTP server. All the messages are stored in the database (Oracle 10) installed in the same box.

These tests were performed with a Dell 8 core box, linux ... etc. Is an enterprise class server.

I think that Mirth really doesn't have any problem with the performance, as 300.000msg/hour is at least 1000 times higher than any rate I've never seen in a production environment.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
A question of volume wmsTrebor General Discussion 1 08-21-2007 06:08 AM


All times are GMT -8. The time now is 07:47 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation