SFTP error due to file size - trying to set up an alert.

[Brad_AZ]

We have some issues with large files and SFTP. While others are looking at a solution, I was trying to get some kind of alert as opposed to just checking the logs. The error I get is:

[2019-12-02 10:05:00,040] ERROR (com.mirth.connect.connectors.file.FileReceiver:43 1): Unable to dispatch message to channel XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX. File: smb://SERVERNAME001P/Files/App/Outbound/Charge_20191202.txt
java.io.IOException: File Charge_20191202.txt is too large. Unable to read files greater than 2147483647 bytes.

I can't get an alert to fire, because (I'm guessing) that this is failing before it gets to the channel. Can I alert on the FileReciever? I'm looking, but can't get that sorted. We are on 3.8.1 with Platinum support and extensions.

(And we are trying to get smaller files sent, but that is a battle in itself)

Thanks!

[kirbykn2]

Just to verify, are you trying to pick files up from an SFTP location? If so, can you set the maximum limit on the file size on the source tab?

[Brad_AZ]

Sorry - we generically call our channels SFTP if they are either sending or picking up files via SFTP.

This is a file pickup that SFTPs to a destination. So this issue is a file reader using smb where it can't ingest the file due to size.

File size is set to 0 min, and Ignore Maximum is selected.

[kirbykn2]

is this a valid file you expect to be able to process? If not, can you set the maximum file size to something appropriate?

Does this file contain a batch of messages that you can parse using the batch feature in Mirth?

Also, review the Mirth logs for errors.

[Brad_AZ]

Originally Posted by kirbykn2

is this a valid file you expect to be able to process? YES If not, can you set the maximum file size to something appropriate?

Does this file contain a batch of messages that you can parse using the batch feature in Mirth NO

Also, review the Mirth logs for errors.

So the file size is a point of contention between the producers of the file and us, but it is a legit file. However, we might set the max to just below what Mirth can accept - then at least we should be able to capture the error at the channel level and alert on it.

The only error I found was the original which included "Unable to read files greater than 2147483647 bytes.". So I will make the max just under that and lobby to have them send us smaller files. Maybe as them to switch to daily as opposed to weekly.

Thanks!~

[agermano]

The issue is that the message needs to be converted to a byte[] at some point, and the max size for an array in Java is Integer.MAX_VALUE (2147483647.)

If it is a delimited file that you can read as a batch, it will actually stream the file and you should be ok since no single message will go over the 2GB mark.

The only other way to stream and split the file would be to process it yourself in a javascript reader.

[jackwhaines]

If you aren't parsing the file (just moving it) you can try setting your Data Types to RAW. That won't load the file into memory at all.

[Brad_AZ]

Thanks for the suggestions. The main issue here was how to alert since it is failing to even get to the channel. I think I have that sorted now by setting max file size to just under the max value pointed out by agermano. This should generate an error and we will be alerted that it did not pass. Before, I found it by seeing the error in the dashboard.

It was already set to RAW, so this error seems to appear earlier on in the process.

Anyway - my task is to get this to alert, and I can do that now. Another team member is chasing the file size issue. If they come up with some solution that is out of the box. I'll do a separate post.

[agermano]

I believe if the file is bigger than the max size it will just get skipped by the File Reader unless you have tested and found otherwise.

I don't know if it is an oversight or what, but Mirth does not generate an event for Alerts to pick up when it encounters that problem. It will generate an error event of type SOURCE_CONNECTOR if it can't read the file or has issues moving or deleting the file.

The error you are seeing will still trigger the error action to allow you to move the file to a different folder (but only after you have downloaded it, and it fails.)