Mirth Community

Mirth Community (http://www.mirthcorp.com/community/forums/index.php)
-   Support (http://www.mirthcorp.com/community/forums/forumdisplay.php?f=6)
-   -   Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog (http://www.mirthcorp.com/community/forums/showthread.php?t=219126)

HGMirth 05-03-2019 11:21 AM

Mirth 3.7 BUG - SFTP downloads slow down causing huge backlog
 
I wanted to post this to the Mirth Jira but the signup link redirects to these forums.

A few weeks ago I upgraded Mirth from 3.6.1 to 3.7.0. Since then, we've had an intermittent issue where an SFTP results download channel would slow down to a crawl causing a huge backlog of unretrieved messages on the remote server.

The only workaround to this has been to completely restart the Mirth service. Changing thread counts, polling intervals, redeploying the channel, etc. have not helped.

I tried rolling back to the previous version by editing the version number in the SCHEMA_INFO table and re-running the 3.6.1 installer, but following that, many of the channels refused to deploy, being reported as invalid, even though as far as I can tell, there were no actual schema changes between the two versions. I ended up having to roll forward, back to 3.7.0.

I'm not sure how to consistently reproduce the issue since the service runs fine for several days before this happens. This is a large instance with hundreds of channels so maybe there's some cumulative effect that triggers this.

Any thoughts on how to fix, roll back or get someone to look at this bug?

agermano 05-08-2019 12:50 PM

In order to roll back, you'll probably want to restore from backup. The channel definitions are stored in a single field in xml. They contain the channel version number and would have been migrated to the newer version when you first deployed them in 3.7, which may make them no longer compatible with the previous version.

Also the forums and Jira share the same user database, so you should be able to log a Jira ticket now that you've created a forum account.

Sorry, I don't have any good suggestions right now for your actual problem.

odo 05-12-2019 11:06 PM

There are changes in the channel structure between 3.6.1 and 3.7. Please find below the (Java) code we use in our migration tool for conversion between both versions:

Code:

        if ((source < 3.7f) && (target >= 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
              // add enabled indicator to transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<script>", "</sequenceNumber>$1<enabled>true</enabled>$1<script>");
                // add enable indicator to filters
                component = component.replaceAll("</sequenceNumber>(\\s*)<operator>", "</sequenceNumber>$1<enabled>true</enabled>$1<operator>");
                // file connector can now define an idle timeout after which the connection is closed. (0 means no timeout - DEFAULT)
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<maxIdleTime>0</maxIdleTime>$2");
              // file connector can now close connection to the file system when not writing
                component = component.replaceAll("(</destinationConnectorProperties>\\s+<scheme>FILE</scheme>.+</timeout>)(\\s*)", "$1$2<keepConnectionOpen>true</keepConnectionOpen>$2");
              // javascript step has now a mirth version attribute (whatsoever...)
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep version=\"" + targetVersion.getVersionString() + "\">");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptrule.JavaScriptRule>", "<com.mirth.connect.plugins.javascriptrule.JavaScriptRule version=\"" + targetVersion.getVersionString() + "\">");
               
                break;
            case <HANDLING OF CODE TEMPLATES && CONTAINERS...>

        } else if ((source >= 3.7f) && (target < 3.7f)) {
            switch (componentType) {
            case MirthHttpsClient.CHANNEL:
                // remove enabled indicator from transformer steps
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<script>", "</sequenceNumber>$1<script>");
               
// remove enabled indicators form filter rules
                component = component.replaceAll("</sequenceNumber>(\\s*)<enabled>[^<]*</enabled>\\s*<operator>", "</sequenceNumber>$1<operator>");
             
// remove "keep connection open" parameter from  file connector
                component = component.replaceAll("</timeout>(\\s*)<keepConnectionOpen>[^<]*</keepConnectionOpen>\\s*<maxIdleTime>", "</timeout>$1<maxIdleTime>");
               
// remove "idle timeout" parameter from file connector
                component = component.replaceAll("</timeout>(\\s*)<maxIdleTime>[^<]*</maxIdleTime>\\s*<secure>", "</timeout>$1<secure>");
               
// remove version attributes from javascript steps and filter rules
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptStep [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptStep>");
                component = component.replaceAll("<com.mirth.connect.plugins.javascriptstep.JavaScriptRule [^>]+>", "<com.mirth.connect.plugins.javascriptstep.JavaScriptRule>");

                break;
            case  <HANDLING OF CODE TEMPLATES && CONTAINERS...>

          }

The green part should be of interest for you. The comments in blue hopefully help to clarify what is going on.

Further, there might be other elements that changed but as the changes are not (publicly) documented we'll add them if they occur.


Finally, the version number of the version attributes also has to be adjusted to the target version. (which should be 3.6.1 in this case)
This is not handled in the shown code segment but can easily be coped with another replaceAll().

narupley 05-31-2019 12:58 PM

Hmm well, the library we use for SFTP is called JSch, and between 3.6 and 3.7 we have not updated that library. Did you make any changes to the version or distribution of Java that you're using?

It would also help to capture a threaddump while the slowdown issue is occurring. You can use "jstack pid", where "pid" is the process ID of your MC server on your operating system.

shaftkb 06-27-2019 10:24 AM

We have the same issue and it occurs about 5 to 6 days after restarting the Mirth Connect service. Upgraded from Mirth 3.4.0 to 3.7.0 with an MSSQL server backend and 200 active channels. The server is used for transmitting HL7 data to/from SFTP sites. We did not upgrade Java when we upgraded Mirth Connect

b33b 07-09-2019 11:10 PM

Quote:

Originally Posted by narupley (Post 266687)
Hmm well, the library we use for SFTP is called JSch, and between 3.6 and 3.7 we have not updated that library. Did you make any changes to the version or distribution of Java that you're using?

It would also help to capture a threaddump while the slowdown issue is occurring. You can use "jstack pid", where "pid" is the process ID of your MC server on your operating system.

We have several sftp connection issues with mirth version >= 3.7.0 & SFTP as well. However I do not think that it might be an issue with Jsch but the way several threads are handled.

We had success on some systems by increasing the polling interval (which is considered to be a workaround and only slows down the occurences of this issue)

On systems with a higher message load (> 1500/hr) it is necessary to restart the mirth service on a regular basis. Processing time is constantly growing from nearly zero (fresh start) to more than 12 seconds (after 4 days)

I guess this is something which has to be looked into for further investigations.

kirbykn2 07-10-2019 06:22 AM

Has anyone updated to 3.8? Does this issue persist in the latest version?


All times are GMT -8. The time now is 10:03 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation