web stats
Large CSV process as batch very slow - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 02-11-2016, 07:36 AM
gonasiri gonasiri is offline
What's HL7?
 
Join Date: Nov 2015
Posts: 1
gonasiri is on a distinguished road
Default Large CSV process as batch very slow

Hi,
I have an interface which reads a csv file and wirted the data to database.
I'm trying to process a 14 MB CSV file as a batch process and it's taking a long time to process this file, as it goes row after row and it has 50,000 rows.

Is there any way to speed up the process?

Thanks
Reply With Quote
  #2  
Old 02-11-2016, 07:42 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,099
narupley is on a distinguished road
Default

Please post the channel here so we can better troubleshoot. First thing is, you could turn on the source queue. Then, does the order in which specific rows are processed by the Database Writer matter? If not, then you can turn on the destination queue and crank up the queue threads. After that to increase throughput further, if you have any source or destination filtering/transformation, you can also move all of that logic into the destination filter/transformer, and then enable "Include Filter/Transformer" on the advanced queue settings.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
Reply

Tags
batch process, csv

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 08:40 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation