web stats
FileReader and "fileLastModified" based filter - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 01-29-2015, 06:31 AM
lionelroux lionelroux is offline
OBX.2 Kenobi
 
Join Date: Apr 2013
Posts: 82
lionelroux is on a distinguished road
Exclamation FileReader and "fileLastModified" based filter

Hello,

I've a channel reading in a directory (using FileReader connector) that contains thousands of DICOM files.
As expected, these DICOM files are heavy (from 20 MB to 1 GB).

In the FileReader connector, I set up a filter to getting files that are older than a defined date (let's says 30 minutes) based on the fileLastModified property.

I have search and read about these feature and it seems it first loads files before testing the fileLastModified properties.

http://www.mirthcorp.com/community/f...ead.php?t=7487

With DICOM files, it's not manageable.

Is there a way to apply such a filter prior to load theses files ?

Otherwise, I would develop a JS connector based on libs like JCIFS, but it's not IMO the best option.

Thx for your help.

PS : I use MC 3.1.1.7461

Last edited by lionelroux; 01-29-2015 at 06:33 AM. Reason: version
Reply With Quote
  #2  
Old 01-29-2015, 08:01 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,126
narupley is on a distinguished road
Default

First, what exactly are you talking about with the "fileLastModified property"? Are you talking about the actual OS last modified timestamp for the file? Or are you talking about a DICOM-specific tag contained within the file?

If you're talking about the OS last modified timestamp, then it's not true that files get loaded into memory before that value is checked. Only the OS file information (like the filename, last modified date, size, etc.) is loaded, but not the file contents.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 01-29-2015, 08:15 AM
lionelroux lionelroux is offline
OBX.2 Kenobi
 
Join Date: Apr 2013
Posts: 82
lionelroux is on a distinguished road
Default

yes I talked about the OS property.
So I'm conforted to know files are not loaded.

So maybe it's because there are so many files in the directory that it takes so much time to read files and send them into my channel.

So i'll apply this filter.

Thank you.
Reply With Quote
  #4  
Old 01-30-2015, 08:03 AM
lionelroux lionelroux is offline
OBX.2 Kenobi
 
Join Date: Apr 2013
Posts: 82
lionelroux is on a distinguished road
Default

Back with my FileReader memory "issue".

So today I tried to load some files in the directory. There are 483 DICOM files in it.
I set up a filter based on the OS last modified property to get only files that are not older than 1 hour. This files subset is approx. 30 files.

My channel has a queue set on for the source connector.

The first 300 files are processed very quickly, let's say 75 files every 10 sec.
But then, it takes more and more time, and at the end, I get an out Of Memory error from the server.

Here is the Stacktrace :

ERROR 2015-01-30 16:20:44,946 [Thread-126] Server: Exception in thread "Thread-126" java.lang.OutOfMemoryError: Java heap space
ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.util.Arrays.copyOf(Arrays.java:2367)
ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:130)
ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(AbstractStringBuilder.java:114)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:535)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.lang.StringBuffer.append(StringBuffer.java:32 2)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.io.StringWriter.write(StringWriter.java:94)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xml.internal.serializer.ToStrea m.characters(ToStream.java:1504)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xml.internal.serializer.ToStrea m.characters(ToStream.java:1618)
ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:244)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:230)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:230)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:136)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:98)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transformIdentity(TransformerImpl.java:
683)
ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(TransformerImpl.java:727)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(TransformerImpl.java:340)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.model.converters.DocumentSeriali zer.toXML(DocumentSerializer.java:94)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.model.converters.DocumentSeriali zer.toXML(DocumentSerializer.java:102)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.plugins.datatypes.dicom.DICOMSer ializer.toXML(DICOMSerializer.java:181)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran
sformerExecutor.java:91)
ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1473)
ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.pr ocessSourceQueue(Channel.java:1738)
ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.ru n(Channel.java:1725)
ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at java.lang.Thread.run(Thread.java:722)


My box is a Linux one, with 6 Go memory, 3Go for the JVM, 4 CPU-cores (all on a VM-ware env.).

So it seems there is something with the memory even if the message is filtered.
Any idea ?

thx in advance
Reply With Quote
  #5  
Old 02-02-2015, 05:47 AM
lionelroux lionelroux is offline
OBX.2 Kenobi
 
Join Date: Apr 2013
Posts: 82
lionelroux is on a distinguished road
Default

The DICOM-to-XML process is the root cause of my OOM error.
If I set datatypes as DICOM, without converting to XML, it's ok.

So the workaround for my case is to set up a child channel that receives only the non-filtered messages. This one will convert DICOm messages to XMl so that I can alter some values.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 02:21 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Mirth Corporation