web stats
Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 05-03-2019, 12:21 PM
HGMirth HGMirth is offline
What's HL7?
 
Join Date: May 2019
Posts: 1
HGMirth is on a distinguished road
Exclamation Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog

I wanted to post this to the Mirth Jira but the signup link redirects to these forums.

A few weeks ago I upgraded Mirth from 3.6.1 to 3.7.0. Since then, we've had an intermittent issue where an SFTP results download channel would slow down to a crawl causing a huge backlog of unretrieved messages on the remote server.

The only workaround to this has been to completely restart the Mirth service. Changing thread counts, polling intervals, redeploying the channel, etc. have not helped.

I tried rolling back to the previous version by editing the version number in the SCHEMA_INFO table and re-running the 3.6.1 installer, but following that, many of the channels refused to deploy, being reported as invalid, even though as far as I can tell, there were no actual schema changes between the two versions. I ended up having to roll forward, back to 3.7.0.

I'm not sure how to consistently reproduce the issue since the service runs fine for several days before this happens. This is a large instance with hundreds of channels so maybe there's some cumulative effect that triggers this.

Any thoughts on how to fix, roll back or get someone to look at this bug?
Reply With Quote
  #2  
Old 05-08-2019, 01:50 PM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 1,028
agermano is on a distinguished road
Default

In order to roll back, you'll probably want to restore from backup. The channel definitions are stored in a single field in xml. They contain the channel version number and would have been migrated to the newer version when you first deployed them in 3.7, which may make them no longer compatible with the previous version.

Also the forums and Jira share the same user database, so you should be able to log a Jira ticket now that you've created a forum account.

Sorry, I don't have any good suggestions right now for your actual problem.
Reply With Quote
  #3  
Old 05-13-2019, 12:06 AM
odo odo is offline
OBX.3 Kenobi
 
Join Date: Feb 2017
Location: Luxembourg
Posts: 144
odo is on a distinguished road
Default

There are changes in the channel structure between 3.6.1 and 3.7. Please find below the (Java) code we use in our migration tool for conversion between both versions:

Code:
        if ((source < 3.7f) && (target >= 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
                // add enabled indicator to transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<script>", "</sequenceNumber>$1<enabled>true</enabled>$1<script>");
                // add enable indicator to filters
                component = component.replaceAll("</sequenceNumber>(\\s*)<operator>", "</sequenceNumber>$1<enabled>true</enabled>$1<operator>");
                // file connector can now define an idle timeout after which the connection is closed. (0 means no timeout - DEFAULT)
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<maxIdleTime>0</maxIdleTime>$2");
                // file connector can now close connection to the file system when not writing
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<keepConnectionOpen>true</keepConnectionOpen>$2");
                // javascript step has now a mirth version attribute (whatsoever...)
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep version=\"" + targetVersion.getVersionString() + "\">");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptrule.JavaScriptRule>", "<com.mirth.connect.plugins.javascriptrule.JavaScriptRule version=\"" + targetVersion.getVersionString() + "\">");
                
                break;
            case <HANDLING OF CODE TEMPLATES && CONTAINERS...>

         } else if ((source >= 3.7f) && (target < 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
                // remove enabled indicator from transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<script>", "</sequenceNumber>$1<script>");
                // remove enabled indicators form filter rules
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<operator>", "</sequenceNumber>$1<operator>");
                // remove "keep connection open" parameter from  file connector
                component = component.replaceAll("</timeout>(\\s*)<keepConnectionOpen>[^<]*</keepConnectionOpen>\\s*<maxIdleTime>", "</timeout>$1<maxIdleTime>");
                // remove "idle timeout" parameter from file connector
                component = component.replaceAll("</timeout>(\\s*)<maxIdleTime>[^<]*</maxIdleTime>\\s*<secure>", "</timeout>$1<secure>");
                // remove version attributes from javascript steps and filter rules
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptRule [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptRule>");

                 break;
            case  <HANDLING OF CODE TEMPLATES && CONTAINERS...>

           }
The green part should be of interest for you. The comments in blue hopefully help to clarify what is going on.

Further, there might be other elements that changed but as the changes are not (publicly) documented we'll add them if they occur.


Finally, the version number of the version attributes also has to be adjusted to the target version. (which should be 3.6.1 in this case)
This is not handled in the shown code segment but can easily be coped with another replaceAll().

Last edited by odo; 05-13-2019 at 12:12 AM.
Reply With Quote
  #4  
Old 05-31-2019, 01:58 PM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

Hmm well, the library we use for SFTP is called JSch, and between 3.6 and 3.7 we have not updated that library. Did you make any changes to the version or distribution of Java that you're using?

It would also help to capture a threaddump while the slowdown issue is occurring. You can use "jstack pid", where "pid" is the process ID of your MC server on your operating system.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #5  
Old 06-27-2019, 11:24 AM
shaftkb shaftkb is offline
Mirth Newb
 
Join Date: Oct 2007
Posts: 10
shaftkb
Default

We have the same issue and it occurs about 5 to 6 days after restarting the Mirth Connect service. Upgraded from Mirth 3.4.0 to 3.7.0 with an MSSQL server backend and 200 active channels. The server is used for transmitting HL7 data to/from SFTP sites. We did not upgrade Java when we upgraded Mirth Connect

Last edited by shaftkb; 06-27-2019 at 11:27 AM.
Reply With Quote
  #6  
Old 07-10-2019, 12:10 AM
b33b b33b is offline
What's HL7?
 
Join Date: Jan 2011
Posts: 5
b33b is on a distinguished road
Default

Quote:
Originally Posted by narupley View Post
Hmm well, the library we use for SFTP is called JSch, and between 3.6 and 3.7 we have not updated that library. Did you make any changes to the version or distribution of Java that you're using?

It would also help to capture a threaddump while the slowdown issue is occurring. You can use "jstack pid", where "pid" is the process ID of your MC server on your operating system.
We have several sftp connection issues with mirth version >= 3.7.0 & SFTP as well. However I do not think that it might be an issue with Jsch but the way several threads are handled.

We had success on some systems by increasing the polling interval (which is considered to be a workaround and only slows down the occurences of this issue)

On systems with a higher message load (> 1500/hr) it is necessary to restart the mirth service on a regular basis. Processing time is constantly growing from nearly zero (fresh start) to more than 12 seconds (after 4 days)

I guess this is something which has to be looked into for further investigations.
Reply With Quote
  #7  
Old 07-10-2019, 07:22 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 586
kirbykn2 is on a distinguished road
Default

Has anyone updated to 3.8? Does this issue persist in the latest version?
__________________
Best,

Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.4
Mirth Connect Version 3.8.0
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_181-b13
PostgreSQL Version 9.6.8
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 08:15 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation