web stats
Heap size error when reading HL7 Batch file to add header and footer segments - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 08-16-2016, 02:55 PM
jjones jjones is offline
 
Join Date: Jan 2016
Location: Blanding, UT
Posts: 19
jjones is on a distinguished road
Default Heap size error when reading HL7 Batch file to add header and footer segments

I am getting a "java.lang.OutOfMemoryError: Java heap space" error when I have a channel that is a file reader that tries to read a file that has 9,000+ HL7 messages in it (size of the file is 1.3MB). I am trying to use this channel to put headers and footer segments around all the messages and count the total number of messages.

I have tried to up the heap size but if I do anything over 512MB then it won't open administrator. The it works find when I have 100 or less messages in the file, but when I get into the thousands, it gives this error.

These are the transformer steps and destination template that I am using.
Transformer steps:
Code:
//Loops to see how many MSH segments are in batch message
var mshCount = 0;

for each (msh in msg.MSH){
	mshCount++;
};

//# value of total messages in batch file
channelMap.put("count", mshCount);

//string value of total messages in batch file
channelMap.put("countstr", mshCount.toString());
Code:
//Returns current date for input into HL7 fields
var curdate = DateUtil.getCurrentDate('yyyyMMddhhmmss');
channelMap.put("Current Date",curdate );
Code:
//Total number of messages in batch file including BHS and ZHS segments
var TotalNumberMessages = ($('count') + 2).toString();
channelMap.put("TotalBatchMessages",TotalNumberMessages);
Code:
//Returns current date for input into HL7 fields
var curdatenotime = DateUtil.getCurrentDate('yyyyMMdd');
channelMap.put("Current Date-notime",curdatenotime );
Destination Template:
Code:
BHS|^~\&|XXXX|XXXXXX|||${maps.get('Current Date')}|||BusinessName
ZHS||
${message.encodedData}
ZTS|0|0|0|0|${TotalBatchMessages}
BTS|${countstr}||1
What can be done to be able to process these messages? I am currently developing another channel that might have batch files that could have more than 20,000 messages in them.

Mirth Version info:
Mirth Connect Server 3.3.1.7856
Built on October 15, 2015
Java version: 1.8.0_91
Reply With Quote
  #2  
Old 08-16-2016, 08:03 PM
joshua_douglas joshua_douglas is offline
Mirth Newb
 
Join Date: Mar 2014
Posts: 16
joshua_douglas is on a distinguished road
Default

Open Mcserver.vmoptions or mcservice.vmoptions depending on your install and increase your heap size:
change this line
-Xmx256m

to whatever you want it to be

ie. ( for 1gb)
-Xmx1024m
Reply With Quote
  #3  
Old 08-17-2016, 03:40 AM
siddharth siddharth is offline
Mirth Guru
 
Join Date: Feb 2013
Posts: 835
siddharth is on a distinguished road
Default

Regardless of increasing the heap size in mirth settings, you are likely to stumble upon Java heap space error again. There is a rendering step that converts the HL7 message in string(message) format to its XML representation (msg) inside the transformer. You have 9000 messages in a single file, it will choke on the heap size.

Now the solution is to use a JS reader, read the file as a string and count number of occurences of MSH, which will eventually give you the count.
Reply With Quote
  #4  
Old 08-17-2016, 08:38 AM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

The File Reader with inbound HL7 v2.x data type supports batching, and will not read the entire file into memory at once. When you enable batching, there are variables in the source map that can help determine where you are in the batch (the message count and whether the current message is the last one in a batch). Unfortunately right now the HL7 batch parser skips over batch segments like BHS/BTS.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #5  
Old 08-19-2016, 11:12 AM
jjones jjones is offline
 
Join Date: Jan 2016
Location: Blanding, UT
Posts: 19
jjones is on a distinguished road
Default Thanks

I have tried the heap size changes but with no luck. I understand what Narupley is saying. I have not figured this out quite yet but will try a few more things to see if I can get this to work. Thank you all for your help and suggestions.
Reply With Quote
  #6  
Old 11-02-2016, 11:00 AM
steve.ela steve.ela is offline
OBX.1 Kenobi
 
Join Date: Nov 2014
Posts: 38
steve.ela is on a distinguished road
Default

I have a project starting to recreate a flat file of updated HL7 messages, with the *original* headers and footers.
Not certain how to either store the headers/footers, or read them again at the time I build the outbound message.

Anyone else come across this scenario?
Reply With Quote
Reply

Tags
heap space, hl7 messages, large files, out of memory error

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 11:16 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation