web stats
Parsing 834 out of memory - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 02-22-2018, 01:15 PM
tkurian tkurian is offline
Mirth Newb
 
Join Date: Jan 2018
Posts: 11
tkurian is on a distinguished road
Default Parsing 834 out of memory

I'm a newbie of EDI trying to parse an 834 file and just simple turn it into an XML file. However Mirth keeps saying it's running out of memory.

The 1st error:
[2018-02-22 17:10:07,974] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
java.lang.OutOfMemoryError: Java heap space

The second:
[2018-02-22 17:10:07,974] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 258): Error processing message in channel EDI_Test (7fe4cec7-8678-4163-bd45-332b12aa53dd).
java.lang.OutOfMemoryError
at com.mirth.connect.donkey.server.channel.Channel.ha ndleDestinationChainThrowable(Channel.java:1772)
at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1718)
at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:354)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)

I'm enclosing the channel definition on this thread. Any advice/help on how to parse an EDI to it's component parts would be great.
Attached Files
File Type: xml EDI_Test.xml (77.7 KB, 2 views)
Reply With Quote
  #2  
Old 02-22-2018, 01:45 PM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,115
narupley is on a distinguished road
Default

How large is the file? It may be that you just have to increase the server-side max heap size. If the file contains large embedded blob data (like an embedded image or PDF) you should consider using an Attachment Handler to extract that before the message processes through the channel. Or if it's a batch file containing many messages within, you may want to consider turning on Process Batch on the source settings. There's an example batch script for EDI/X12 on the public Slack group.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 02-23-2018, 05:58 AM
tkurian tkurian is offline
Mirth Newb
 
Join Date: Jan 2018
Posts: 11
tkurian is on a distinguished road
Default Tried to raise Heap Size

Thanks for your reply. The file size is just 7 MB. (I added the new channel definition)

I increased the heap size to 2 GB, I also set the channel to do batching and took a script that I think is the right one from the slack site and I still get this error:

[2018-02-23 09:51:21,169] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at java.io.StringWriter.write(Unknown Source)
at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:74)
at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:113)
at com.mirth.connect.plugins.datatypes.edi.EDIReader. parse(EDIReader.java:154)
at com.mirth.connect.plugins.datatypes.edi.EDISeriali zer.toXML(EDISerializer.java:96)
at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran sformerExecutor.java:93)
at com.mirth.connect.donkey.server.channel.Destinatio nConnector.transform(DestinationConnector.java:360 )
at com.mirth.connect.donkey.server.channel.Destinatio nChain.doCall(DestinationChain.java:114)
at com.mirth.connect.donkey.server.channel.Destinatio nChain.call(DestinationChain.java:63)
at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1716)
at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchBatchMessage(SourceConnector.java:25 4)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchBatchMessage(SourceConnector.java:19 6)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:331)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)
Attached Files
File Type: xml EDI_Test_V2.xml (78.9 KB, 2 views)

Last edited by tkurian; 02-23-2018 at 06:01 AM. Reason: Adding New Channel Definition
Reply With Quote
  #4  
Old 02-23-2018, 07:23 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,115
narupley is on a distinguished road
Default

That must mean the file is not actually a batch of multiple messages, but instead one big message right?

Is there any large blob content in the message that would make sense to extract as an attachment?

Also just to confirm, how exactly did you increase the heap size? If you're launching the service/daemon, did you edit mcservice.vmoptions? If you're launching mcserver directly, did you edit mcserver.vmoptions? Or did you use the Server Manager? Or are you passing in command line options to Java?

I also noticed you're doing this in the destination transformer:



Was that just testing, or else why is that there? The "message" variable is not available from transformers to start with. And then there's no need to serialize the EDI/X12 to XML since the transformer will have already done that for you... the "msg" variable is your E4X XML object. Finally, you're taking that XML and putting it into the channel map, needlessly double-storing the entire message.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.

Last edited by narupley; 02-23-2018 at 07:28 AM.
Reply With Quote
  #5  
Old 02-23-2018, 08:53 AM
tkurian tkurian is offline
Mirth Newb
 
Join Date: Jan 2018
Posts: 11
tkurian is on a distinguished road
Default Next Attempt

The message is an 834 EDI Enrollment file. The message allows for a repeatable section (INS segments) for individual enrollments. So yes it's one EDI file with multiple enrollments. There are no blobs, attachments or anything other than pure EDI in the file.

Yes the destination transformer was for testing and I forgot to remove it. I have removed it now. (I've attached the new channel definition)

I have also removed the batching and script this time and tried to drop the message. When I did this it didn't error out but just put out the same message I got from the source. In the hope of doing the basic I switch the outbound datatype to XML to see if I can just spit out EDI --> XML but I still get the infamous memory error again.

I just have one basic question:

How do I split this 834 to different individual enrollments (members - EDI INS segments) so that I can store each one in the database?


[2018-02-23 12:05:09,250] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at java.io.StringWriter.write(Unknown Source)
at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:74)
at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:113)
at com.mirth.connect.plugins.datatypes.edi.EDIReader. parse(EDIReader.java:154)
at com.mirth.connect.plugins.datatypes.edi.EDISeriali zer.toXML(EDISerializer.java:96)
at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran sformerExecutor.java:93)
at com.mirth.connect.donkey.server.channel.Destinatio nConnector.transform(DestinationConnector.java:360 )
at com.mirth.connect.donkey.server.channel.Destinatio nChain.doCall(DestinationChain.java:114)
at com.mirth.connect.donkey.server.channel.Destinatio nChain.call(DestinationChain.java:63)
at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1716)
at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:354)
at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)
Attached Files
File Type: xml EDI_Test_V3.xml (77.1 KB, 2 views)
Reply With Quote
  #6  
Old 02-23-2018, 10:13 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,115
narupley is on a distinguished road
Default

It sounds like what you want to do is modify that sample batch script so that it splits by enrollment. Right now it just splits by ISA segment.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #7  
Old 02-23-2018, 12:31 PM
tkurian tkurian is offline
Mirth Newb
 
Join Date: Jan 2018
Posts: 11
tkurian is on a distinguished road
Default Thanks

As you said I changed it to INS. I think now it's debatching it without errors not 100% in the right way but I think I'll take a look at the script a little closer and figure out why it's not putting 1 file for each enrollment.

However it's still putting everything out in EDI format is there a way that I can dump it to the folder in XML format?

Thanks for all your help
Attached Files
File Type: xml EDI_Test_V4.xml (78.5 KB, 10 views)
Reply With Quote
  #8  
Old 02-23-2018, 03:00 PM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,115
narupley is on a distinguished road
Default

If you set your destination outbound data type to XML, the encoded data should automatically be converted to XML for you.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #9  
Old 02-27-2018, 05:05 AM
tkurian tkurian is offline
Mirth Newb
 
Join Date: Jan 2018
Posts: 11
tkurian is on a distinguished road
Default Thanks

Thanks it worked great!
Reply With Quote
Reply

Tags
edi

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 01:39 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation