Mirth Connect
  1. Mirth Connect
  2. MIRTH-1441

File Reader - add support for traversing subdirectories

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0 RC 1, 3.0.0
    • Component/s: Administrator, Server
    • Labels:
      None
    1. threadDump_1.txt
      32 kB
      Nick Taylor
    2. threadDump_2.txt
      27 kB
      Nick Taylor

      Activity

      Hide
      Gerald Bortis added a comment -

      The method to load files in the API for the file methods will need to be updated to accept a boolean for recursively traversing directories. Can the existing wildcard or RegEx pattern support be extended to support this?

      Show
      Gerald Bortis added a comment - The method to load files in the API for the file methods will need to be updated to accept a boolean for recursively traversing directories. Can the existing wildcard or RegEx pattern support be extended to support this?
      Hide
      Christopher Ro added a comment -

      Using Java, there's no wildcard or RegEx pattern support for searching through subdirectories recursively. We'll have to add a new component for the boolean parameter.

      Show
      Christopher Ro added a comment - Using Java, there's no wildcard or RegEx pattern support for searching through subdirectories recursively. We'll have to add a new component for the boolean parameter.
      Hide
      Mitch Dempsey added a comment -

      This would require some substantial changes to our API, so we are considering switching to Apache VFS

      Show
      Mitch Dempsey added a comment - This would require some substantial changes to our API, so we are considering switching to Apache VFS
      Hide
      Nick Rupley added a comment -

      Revision 6476: Added support to the File Reader for recursive subdirectory traversal. If recursion is on, FileReceiver keeps track of a set of visited directories so that the same directory would never be read twice (due to symlinks, etc.). The traversal is done in a pre-order, depth-first manner. A dialog has been added to the UI when recursion is enabled to warn users of the implications. In order to facilitate the set of visited directories, a getCanonicalPath method has been added to FileInfo, so that FileSystemConnection implementations have a way to retrieve an absolute and unique path string for the file.

      Show
      Nick Rupley added a comment - Revision 6476: Added support to the File Reader for recursive subdirectory traversal. If recursion is on, FileReceiver keeps track of a set of visited directories so that the same directory would never be read twice (due to symlinks, etc.). The traversal is done in a pre-order, depth-first manner. A dialog has been added to the UI when recursion is enabled to warn users of the implications. In order to facilitate the set of visited directories, a getCanonicalPath method has been added to FileInfo, so that FileSystemConnection implementations have a way to retrieve an absolute and unique path string for the file.
      Hide
      Matt Shigekawa added a comment -

      Verified new "Include All Subdirectories" option in File Reader processes all files in the specified read directory along with its subdirectories.

      Show
      Matt Shigekawa added a comment - Verified new "Include All Subdirectories" option in File Reader processes all files in the specified read directory along with its subdirectories.
      Hide
      Nick Taylor added a comment -

      Are there currently issues with this functionality in Version 3.5.1? I'm trying to use the "Include All Subdirectories" setting on my sftp File Reader channel, but as soon as I turn it on and Redeploy the channel, the Connection stays in a Polling Status and never generates an error or gives any notification something wrong, and I usually have to Halt the job when I try to revert the setting and redeploy.
      When I Halt the job I'll get a email notification stating "Returned object not currently part of this pool". The main directory I am pointed to only has 7 Subfolders directly below. And there is currently a total of 80 files within all the directories combined.
      I looked in the Mirth.log file, and each time I Halt the channel it generates the error below, but does not log anything before or after.
      Is there currently a known bug, or am I probably just not setting something up correctly?
      Thanks.

      ERROR 2018-03-08 19:08:05,500 [File Reader Polling Thread on Passport 277 In - IH (33110d3b-abb0-4ff0-93f6-2ef0f7763d05) < 33110d3b-abb0-4ff0-93f6-2ef0f7763d05_Worker-1] com.mirth.connect.connectors.file.FileReceiver: Error polling in channel: 33110d3b-abb0-4ff0-93f6-2ef0f7763d05
      java.lang.IllegalStateException: Returned object not currently part of this pool
      at org.apache.commons.pool2.impl.GenericObjectPool.returnObject(GenericObjectPool.java:541)
      at com.mirth.connect.connectors.file.FileConnector.releaseConnection(FileConnector.java:205)
      at com.mirth.connect.connectors.file.FileReceiver.listDirectories(FileReceiver.java:574)
      at com.mirth.connect.connectors.file.FileReceiver.listFilesRecursively(FileReceiver.java:222)
      at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:199)
      at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

      Show
      Nick Taylor added a comment - Are there currently issues with this functionality in Version 3.5.1? I'm trying to use the "Include All Subdirectories" setting on my sftp File Reader channel, but as soon as I turn it on and Redeploy the channel, the Connection stays in a Polling Status and never generates an error or gives any notification something wrong, and I usually have to Halt the job when I try to revert the setting and redeploy. When I Halt the job I'll get a email notification stating "Returned object not currently part of this pool". The main directory I am pointed to only has 7 Subfolders directly below. And there is currently a total of 80 files within all the directories combined. I looked in the Mirth.log file, and each time I Halt the channel it generates the error below, but does not log anything before or after. Is there currently a known bug, or am I probably just not setting something up correctly? Thanks. ERROR 2018-03-08 19:08:05,500 [File Reader Polling Thread on Passport 277 In - IH (33110d3b-abb0-4ff0-93f6-2ef0f7763d05) < 33110d3b-abb0-4ff0-93f6-2ef0f7763d05_Worker-1] com.mirth.connect.connectors.file.FileReceiver: Error polling in channel: 33110d3b-abb0-4ff0-93f6-2ef0f7763d05 java.lang.IllegalStateException: Returned object not currently part of this pool at org.apache.commons.pool2.impl.GenericObjectPool.returnObject(GenericObjectPool.java:541) at com.mirth.connect.connectors.file.FileConnector.releaseConnection(FileConnector.java:205) at com.mirth.connect.connectors.file.FileReceiver.listDirectories(FileReceiver.java:574) at com.mirth.connect.connectors.file.FileReceiver.listFilesRecursively(FileReceiver.java:222) at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:199) at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
      Hide
      Nick Rupley added a comment -

      Can you take a threaddump (with jstack) while it's stuck in Polling, and post it here?

      Show
      Nick Rupley added a comment - Can you take a threaddump (with jstack) while it's stuck in Polling, and post it here?
      Hide
      Nick Taylor added a comment -

      Requested files.

      Show
      Nick Taylor added a comment - Requested files.
      Hide
      Nick Taylor added a comment -

      I've attached the dump files. It's the first time I've use jstack, so if it looks like I didn't do it correctly or any information is missing, please let me know. Thanks.

      Show
      Nick Taylor added a comment - I've attached the dump files. It's the first time I've use jstack, so if it looks like I didn't do it correctly or any information is missing, please let me know. Thanks.

        People

        • Assignee:
          Nick Rupley
          Reporter:
          Christopher Ro
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development