web stats
Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 05-03-2019, 11:21 AM
HGMirth HGMirth is offline
What's HL7?
 
Join Date: May 2019
Posts: 1
HGMirth is on a distinguished road
Exclamation Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog

I wanted to post this to the Mirth Jira but the signup link redirects to these forums.

A few weeks ago I upgraded Mirth from 3.6.1 to 3.7.0. Since then, we've had an intermittent issue where an SFTP results download channel would slow down to a crawl causing a huge backlog of unretrieved messages on the remote server.

The only workaround to this has been to completely restart the Mirth service. Changing thread counts, polling intervals, redeploying the channel, etc. have not helped.

I tried rolling back to the previous version by editing the version number in the SCHEMA_INFO table and re-running the 3.6.1 installer, but following that, many of the channels refused to deploy, being reported as invalid, even though as far as I can tell, there were no actual schema changes between the two versions. I ended up having to roll forward, back to 3.7.0.

I'm not sure how to consistently reproduce the issue since the service runs fine for several days before this happens. This is a large instance with hundreds of channels so maybe there's some cumulative effect that triggers this.

Any thoughts on how to fix, roll back or get someone to look at this bug?
Reply With Quote
  #2  
Old 05-08-2019, 12:50 PM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 754
agermano is on a distinguished road
Default

In order to roll back, you'll probably want to restore from backup. The channel definitions are stored in a single field in xml. They contain the channel version number and would have been migrated to the newer version when you first deployed them in 3.7, which may make them no longer compatible with the previous version.

Also the forums and Jira share the same user database, so you should be able to log a Jira ticket now that you've created a forum account.

Sorry, I don't have any good suggestions right now for your actual problem.
Reply With Quote
  #3  
Old 05-12-2019, 11:06 PM
odo odo is offline
OBX.3 Kenobi
 
Join Date: Feb 2017
Location: Luxembourg
Posts: 137
odo is on a distinguished road
Default

There are changes in the channel structure between 3.6.1 and 3.7. Please find below the (Java) code we use in our migration tool for conversion between both versions:

Code:
        if ((source < 3.7f) && (target >= 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
                // add enabled indicator to transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<script>", "</sequenceNumber>$1<enabled>true</enabled>$1<script>");
                // add enable indicator to filters
                component = component.replaceAll("</sequenceNumber>(\\s*)<operator>", "</sequenceNumber>$1<enabled>true</enabled>$1<operator>");
                // file connector can now define an idle timeout after which the connection is closed. (0 means no timeout - DEFAULT)
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<maxIdleTime>0</maxIdleTime>$2");
                // file connector can now close connection to the file system when not writing
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<keepConnectionOpen>true</keepConnectionOpen>$2");
                // javascript step has now a mirth version attribute (whatsoever...)
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep version=\"" + targetVersion.getVersionString() + "\">");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptrule.JavaScriptRule>", "<com.mirth.connect.plugins.javascriptrule.JavaScriptRule version=\"" + targetVersion.getVersionString() + "\">");
                
                break;
            case <HANDLING OF CODE TEMPLATES && CONTAINERS...>

         } else if ((source >= 3.7f) && (target < 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
                // remove enabled indicator from transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<script>", "</sequenceNumber>$1<script>");
                // remove enabled indicators form filter rules
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<operator>", "</sequenceNumber>$1<operator>");
                // remove "keep connection open" parameter from  file connector
                component = component.replaceAll("</timeout>(\\s*)<keepConnectionOpen>[^<]*</keepConnectionOpen>\\s*<maxIdleTime>", "</timeout>$1<maxIdleTime>");
                // remove "idle timeout" parameter from file connector
                component = component.replaceAll("</timeout>(\\s*)<maxIdleTime>[^<]*</maxIdleTime>\\s*<secure>", "</timeout>$1<secure>");
                // remove version attributes from javascript steps and filter rules
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptRule [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptRule>");

                 break;
            case  <HANDLING OF CODE TEMPLATES && CONTAINERS...>

           }
The green part should be of interest for you. The comments in blue hopefully help to clarify what is going on.

Further, there might be other elements that changed but as the changes are not (publicly) documented we'll add them if they occur.


Finally, the version number of the version attributes also has to be adjusted to the target version. (which should be 3.6.1 in this case)
This is not handled in the shown code segment but can easily be coped with another replaceAll().

Last edited by odo; 05-12-2019 at 11:12 PM.
Reply With Quote
  #4  
Old 05-31-2019, 12:58 PM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,098
narupley is on a distinguished road
Default

Hmm well, the library we use for SFTP is called JSch, and between 3.6 and 3.7 we have not updated that library. Did you make any changes to the version or distribution of Java that you're using?

It would also help to capture a threaddump while the slowdown issue is occurring. You can use "jstack pid", where "pid" is the process ID of your MC server on your operating system.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 05:54 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation