web stats
memory usage filereader - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 05-28-2015, 07:57 AM
Jurjan Jurjan is offline
OBX.1 Kenobi
 
Join Date: Feb 2011
Posts: 37
Jurjan is on a distinguished road
Default memory usage filereader

Since we migrated to 3.2.1 (from 2.X.X) we've seen that memory usage has increased, to the point that even 6GB isn't enough anymore.

We're getting the infamous out of memory error for the heapspace.

now, we DO read in (filereader source) files of up to about 200MB.
and use a filewriter destination to another directory on the same system (with renaming etc of file).

This reading seems to result in memory usage of far more than just the 200MB of the file itself.
and what's worse, it seems like that memory isn't freed up after it's been used.
resulting in out of memory.

We have a channel that call system.gc(), but that doesn't have an effect.

I've used a testchannel to read in these files using javascreipt souces.
using Packages.java.io.File listfiles
and java.lang.Runtime.getRuntime().exec("cmd.exe /c move from_file to_file")

this seems to work without all the overhead, but is less easy to maintain etc.

is there some way of using a filereader filewriter solution that doesn't use so much memory?

Thanks!
Reply With Quote
  #2  
Old 05-28-2015, 08:28 AM
upstart33 upstart33 is offline
Mirth Guru
 
Join Date: Dec 2010
Location: Chicago, IL.
Posts: 460
upstart33 is on a distinguished road
Default

Unfortunately, as far as I understand, Mirth keeps several different versions/copies of each message that is processed through a channel. So, if you read in 1 200 MB file, the memory impact is actually much higher as it stores a Raw, Transformed, and Encoded version of the message.
Reply With Quote
  #3  
Old 05-28-2015, 11:30 AM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

Mirth Connect 3.0 has additional features that require more memory usage than 2.x did.

The message data is copied in memory at various points to allow for message transformations, so if you do not need to do any transformations on that 200MB, it is better to extract it as an attachment using an Attachment Handler (in the Channel Properties section when editing a channel).

The attachment handler will extract the data when the message is received, before the pre-processor or any transformers are run, then will automatically re-attach on send. This will minimize the amount of memory usage.
Reply With Quote
  #4  
Old 05-28-2015, 11:35 PM
Jurjan Jurjan is offline
OBX.1 Kenobi
 
Join Date: Feb 2011
Posts: 37
Jurjan is on a distinguished road
Default

@Upstart33,
yeah, that's what I expected.
However, I can live with that, as long as that memory gets freed as soon as possible.
Anecdotal evidence seems to suggest to me that that doesn't happen, and that therefore memory full errors are a matter of waiting.

@Brentm,
Sounds interesting.
so how should I work this?
The channel in question does a filereader of a specific directory.
Renames some of the files found.
Moves all of them to another directory.

I'm not seeing where this attachment setting enters the scene.
Please feel free to answer: RTFM.

Thanks!
Reply With Quote
  #5  
Old 05-29-2015, 05:15 AM
rdejournett rdejournett is offline
OBX.2 Kenobi
 
Join Date: Jan 2013
Posts: 99
rdejournett is on a distinguished road
Default

I had similar issues with much smaller files, unfortunately I actually process those files so I need to split it out. My issue was with reprocessing, even displaying the message would cause the out of memory error. I was not able to get it resolved unfortunately.

The only thing that I could suggest is to split the fie as soon as possible if it is a batch file, minimize/eliminate transformations, maybe even reduce the message storage slider (i'm not sure what that would do for memory consumption, but it would help with database usage if I understand correctly).
Reply With Quote
  #6  
Old 05-29-2015, 05:26 AM
Jurjan Jurjan is offline
OBX.1 Kenobi
 
Join Date: Feb 2011
Posts: 37
Jurjan is on a distinguished road
Default

rdejournett,
it's even worse for us, we are in this case not even interested in the data itself, since we are only interested in moving the file from one directory to another.

possibly we'll just start using bat files etc which we wil start using Mirth.

Thanks for replying.
Reply With Quote
  #7  
Old 05-29-2015, 08:01 AM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

Quote:
Sounds interesting.
so how should I work this?
Since you aren't doing any transformations on the content, you can extract the entire contents of each file as the attachment. This means as soon as the file is read, the content will be extracted and excluded from the transformation steps, which is what duplicates the content in memory. When the file is written back out it will re-attach the content immediately before writing. That's how it keeps the memory usage to a minimum.

The easiest way to do that would be to use a JavaScript attachment handler with the following one-liner:
Code:
return addAttachment(message, "text/plain").getAttachmentId();
See attached screenshot
Attached Images
File Type: png atthandler.png (25.1 KB, 34 views)
Reply With Quote
  #8  
Old 06-01-2015, 12:12 AM
Jurjan Jurjan is offline
OBX.1 Kenobi
 
Join Date: Feb 2011
Posts: 37
Jurjan is on a distinguished road
Default

@Brentm,
that works, on our system it takes less memory AND no problems wit memory overflow), thanks.

In the userguide (3.1, october 2014) it didn't make it clear for me how attachments work, this goes quite a ways to do that.
Is there somewhere I can read some more (including examples if possible) about these (for me) obscure parts of MirthConnect

We'll try this out on our clients installation soon, and see where that takes us.

Thanks to all responders.

Jurjan
Reply With Quote
  #9  
Old 06-01-2015, 08:25 AM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

Quote:
Is there somewhere I can read some more (including examples if possible) about these (for me) obscure parts of MirthConnect
For non-subscribers / open-source users, the resources you have available are:

More resources are available with a paid subscription.

In many cases, just searching the forums will get you the info you need. The user guide does give an overview of the attachment handlers, however it's not clear that attachment handlers are the way to resolve memory issues with large messages. I'll make a note to clarify that in future revisions of the user guide.
Reply With Quote
  #10  
Old 06-01-2015, 11:32 PM
Jurjan Jurjan is offline
OBX.1 Kenobi
 
Join Date: Feb 2011
Posts: 37
Jurjan is on a distinguished road
Default

Brentm,
once again, thanks.

Jurjan
Reply With Quote
Reply

Tags
free, heap space, java, memory

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 05:33 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation