web stats
SFTP error due to file size - trying to set up an alert. - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 12-02-2019, 09:14 AM
Brad_AZ Brad_AZ is offline
Mirth Newb
 
Join Date: Mar 2018
Location: Arizona, USA
Posts: 23
Brad_AZ is on a distinguished road
Default SFTP error due to file size - trying to set up an alert.

We have some issues with large files and SFTP. While others are looking at a solution, I was trying to get some kind of alert as opposed to just checking the logs. The error I get is:

Quote:
[2019-12-02 10:05:00,040] ERROR (com.mirth.connect.connectors.file.FileReceiver:43 1): Unable to dispatch message to channel XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX. File: smb://SERVERNAME001P/Files/App/Outbound/Charge_20191202.txt
java.io.IOException: File Charge_20191202.txt is too large. Unable to read files greater than 2147483647 bytes.
I can't get an alert to fire, because (I'm guessing) that this is failing before it gets to the channel. Can I alert on the FileReciever? I'm looking, but can't get that sorted. We are on 3.8.1 with Platinum support and extensions.

(And we are trying to get smaller files sent, but that is a battle in itself)

Thanks!
__________________
SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America

Last edited by Brad_AZ; 12-02-2019 at 09:17 AM.
Reply With Quote
  #2  
Old 12-03-2019, 07:31 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 586
kirbykn2 is on a distinguished road
Default

Just to verify, are you trying to pick files up from an SFTP location? If so, can you set the maximum limit on the file size on the source tab?
__________________
Best,

Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.4
Mirth Connect Version 3.8.0
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_181-b13
PostgreSQL Version 9.6.8
Reply With Quote
  #3  
Old 12-03-2019, 09:00 AM
Brad_AZ Brad_AZ is offline
Mirth Newb
 
Join Date: Mar 2018
Location: Arizona, USA
Posts: 23
Brad_AZ is on a distinguished road
Default

Sorry - we generically call our channels SFTP if they are either sending or picking up files via SFTP.

This is a file pickup that SFTPs to a destination. So this issue is a file reader using smb where it can't ingest the file due to size.

File size is set to 0 min, and Ignore Maximum is selected.
__________________
SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America
Reply With Quote
  #4  
Old 12-03-2019, 09:06 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 586
kirbykn2 is on a distinguished road
Default

is this a valid file you expect to be able to process? If not, can you set the maximum file size to something appropriate?

Does this file contain a batch of messages that you can parse using the batch feature in Mirth?

Also, review the Mirth logs for errors.
__________________
Best,

Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.4
Mirth Connect Version 3.8.0
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_181-b13
PostgreSQL Version 9.6.8
Reply With Quote
  #5  
Old 12-03-2019, 09:45 AM
Brad_AZ Brad_AZ is offline
Mirth Newb
 
Join Date: Mar 2018
Location: Arizona, USA
Posts: 23
Brad_AZ is on a distinguished road
Default

Quote:
Originally Posted by kirbykn2 View Post
is this a valid file you expect to be able to process? YES If not, can you set the maximum file size to something appropriate?

Does this file contain a batch of messages that you can parse using the batch feature in Mirth NO

Also, review the Mirth logs for errors.
So the file size is a point of contention between the producers of the file and us, but it is a legit file. However, we might set the max to just below what Mirth can accept - then at least we should be able to capture the error at the channel level and alert on it.

The only error I found was the original which included "Unable to read files greater than 2147483647 bytes.". So I will make the max just under that and lobby to have them send us smaller files. Maybe as them to switch to daily as opposed to weekly.

Thanks!~
__________________
SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America
Reply With Quote
  #6  
Old 12-03-2019, 11:02 AM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 1,028
agermano is on a distinguished road
Default

The issue is that the message needs to be converted to a byte[] at some point, and the max size for an array in Java is Integer.MAX_VALUE (2147483647.)

If it is a delimited file that you can read as a batch, it will actually stream the file and you should be ok since no single message will go over the 2GB mark.

The only other way to stream and split the file would be to process it yourself in a javascript reader.
Reply With Quote
  #7  
Old 12-03-2019, 11:31 AM
jackwhaines jackwhaines is offline
 
Join Date: May 2011
Location: Kansas City, Missouri
Posts: 246
jackwhaines is on a distinguished road
Send a message via Skype™ to jackwhaines
Default

If you aren't parsing the file (just moving it) you can try setting your Data Types to RAW. That won't load the file into memory at all.
__________________

-= Jack Haines : Founder/CEO of Healthcare Integrations, LLC
-= jack.haines@HealthcareIntegrations.com
-= Mirth Connect (Advanced)-certified
-= Gold member of HL7.org
-= Available for Mirth Connect channel development and consultation! Schedule a FREE call with me at https://calendly.com/jackhaines
Reply With Quote
  #8  
Old 12-03-2019, 12:02 PM
Brad_AZ Brad_AZ is offline
Mirth Newb
 
Join Date: Mar 2018
Location: Arizona, USA
Posts: 23
Brad_AZ is on a distinguished road
Default

Thanks for the suggestions. The main issue here was how to alert since it is failing to even get to the channel. I think I have that sorted now by setting max file size to just under the max value pointed out by agermano. This should generate an error and we will be alerted that it did not pass. Before, I found it by seeing the error in the dashboard.

It was already set to RAW, so this error seems to appear earlier on in the process.

Anyway - my task is to get this to alert, and I can do that now. Another team member is chasing the file size issue. If they come up with some solution that is out of the box. I'll do a separate post.
__________________
SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America
Reply With Quote
  #9  
Old 12-03-2019, 04:37 PM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 1,028
agermano is on a distinguished road
Default

I believe if the file is bigger than the max size it will just get skipped by the File Reader unless you have tested and found otherwise.

I don't know if it is an oversight or what, but Mirth does not generate an event for Alerts to pick up when it encounters that problem. It will generate an error event of type SOURCE_CONNECTOR if it can't read the file or has issues moving or deleting the file.

The error you are seeing will still trigger the error action to allow you to move the file to a different folder (but only after you have downloaded it, and it fails.)
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 12:50 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation