web stats
How to perform actions after a File Reader poll is complete? - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 03-07-2018, 09:10 AM
jridderhoff jridderhoff is offline
OBX.1 Kenobi
 
Join Date: Jan 2015
Posts: 37
jridderhoff is on a distinguished road
Default How to perform actions after a File Reader poll is complete?

This seems like something that shouldn't be hard to do, so I'm either being blind & missing an option, or just being stupid and not thinking of something obvious.

I have a channel with a source type of File Reader. This channel runs hourly to poll for all files in one directory and move them to another directory. After it's completed processing all of the files, I need to change the value of some global variables (not after each file, but only after all of the files in the current poll have been processed).

If it were a Database Reader source, there's an obvious "After all messages" option to run script, but I don't see the same for File Reader. Is there some trick to accomplishing this same functionality via a transformer or JavaScript Writer destination?
Reply With Quote
  #2  
Old 03-07-2018, 11:44 AM
appsbyaaron's Avatar
appsbyaaron appsbyaaron is offline
Mirth Guru
 
Join Date: Nov 2011
Location: Florida, USA
Posts: 377
appsbyaaron is on a distinguished road
Default

Personally I would run it all in a Source Transformer javascript. Then you know when files are done being written and even check to make sure. Then you can change all the variables you want from there.
Reply With Quote
  #3  
Old 03-07-2018, 01:45 PM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 967
agermano is on a distinguished road
Default

I don't think there is a way to do that with a file reader.

You could do it with a javascript reader, but you'd basically need to load all of your files from that poll into memory, and I'm not sure if that's feasible. You could set a sourceMap variable on the last one as an indicator, and then check for the indicator in the response map of the destination or the post processor to know when the last one has been processed.

If you share more about why you need to set a global variable after the polling batch is complete, maybe there is an alternate solution that is less messy.
Reply With Quote
  #4  
Old 03-07-2018, 03:57 PM
jridderhoff jridderhoff is offline
OBX.1 Kenobi
 
Join Date: Jan 2015
Posts: 37
jridderhoff is on a distinguished road
Default

Quote:
Originally Posted by agermano View Post
If you share more about why you need to set a global variable after the polling batch is complete, maybe there is an alternate solution that is less messy.
We've recently started picking up a LOT more volume of ADT messages from certain senders. We have to do a batch send of those ADT messages to a recipient hourly at the most frequent. I had a multi-channel solution in place to assemble the batch message that had been working okay for awhile until the volume increase. In order to avoid running the heap out of memory, we had to cap the batches to 3,000 ADT messages per hourly run (and we're using all we are allotted on the server resources already).

Now, we're getting enough messages that batching 3,000 messages hourly, 24 times a day, isn't enough to keep up, so I've tried working on this solution that will poll hourly, find all files received in that past hour, append them to a file on the file system (which doesn't cause the heap to run away with itself), and then my goal would be, after that process is done, use a 2nd channel (which would only run if the global variable flag is set) to move the assembled file to a location where it can be picked up via a cron'ned shell script to SFTP the files to the recipient.

I don't want the batch file to be assembled in the directory where the cron script will pick it up, because it might run while the poller is still appending files to it, so I need to assemble it in a temp location and then move it only when the poll is finished.

Last edited by jridderhoff; 03-07-2018 at 04:03 PM.
Reply With Quote
  #5  
Old 03-07-2018, 10:13 PM
aTom aTom is offline
OBX.3 Kenobi
 
Join Date: Feb 2016
Location: Lake Geneva Area
Posts: 113
aTom is on a distinguished road
Default

Hello,

Quote:
Originally Posted by jridderhoff View Post
I don't want the batch file to be assembled in the directory where the cron script will pick it up, because it might run while the poller is still appending files to it, so I need to assemble it in a temp location and then move it only when the poll is finished.
Actually you could - without any programming.

You can append incoming messages into a file named {DATE.get('YYYYMMDDHH')}.hl7, then set up other channel to execute hourly 5 after the hour and not to pick up files that were last modified in the last (let's say) 3-4 minutes.

Another way to do this is to pick up individual messages without deleting them (set After processing action to None). Then in your source transformer get a File object from $('originalFilename') and $('fileDirectory') and read its properties. Finally you can delete relevant files using FileUtil.delete() method.
__________________
Tom
Reply With Quote
  #6  
Old 03-08-2018, 07:25 AM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 967
agermano is on a distinguished road
Default

Are these files being moved around on the local file system? Are you doing any processing (reading or changing values) of the hl7 as they pass through? If you're strictly dealing with local files where you don't care about the content there are much faster and more memory/cpu/db efficient ways to move them than treating them as individual messages.
Reply With Quote
  #7  
Old 03-08-2018, 08:31 AM
jridderhoff jridderhoff is offline
OBX.1 Kenobi
 
Join Date: Jan 2015
Posts: 37
jridderhoff is on a distinguished road
Default

Quote:
Originally Posted by agermano View Post
Are these files being moved around on the local file system? Are you doing any processing (reading or changing values) of the hl7 as they pass through? If you're strictly dealing with local files where you don't care about the content there are much faster and more memory/cpu/db efficient ways to move them than treating them as individual messages.
Nope, no processing of the individual messages themselves: they're literally being received from several senders, and then hourly batched together & sent to another recipient. None of the messages individually need to be loaded into Mirth, just moved around on the local file system.

The only special processing we have to do is add the FHS/BHS & BTS/FTS segments to the batched message before it goes out, and name the file to a given standard that the recipient requires. Other than that, we don't do a thing with them.
Reply With Quote
  #8  
Old 03-08-2018, 03:41 PM
jridderhoff jridderhoff is offline
OBX.1 Kenobi
 
Join Date: Jan 2015
Posts: 37
jridderhoff is on a distinguished road
Lightbulb Implemented solution with JavaScript Reader

Just to follow up, I was able to accomplish what I needed with a bit of custom FileUtils work in a JavaScript Reader source. There may be some ways to min/max this & make it a bit more efficient, but it's a vast improvement over the old way (tried with up to 10K messages, each ~1KB in size, and it handled it fine in about 10 seconds).

The files to send get dropped into /path/to/files/in by another process, and then this channel assembles the batched message & moves it into the /path/to/cron folder where a cron job picks up the file & transmits it to the recipient via sFTP.

Hopefully it'll help someone else, and I don't mind any kibbutzing either, if there's a more elegant/performant way to do this.

Thanks all for the input!

JavaScript Reader Source:
Code:
var pollDirectory = new java.io.File("/path/to/files/in/");
var processedDirectory = new java.io.File("/path/to/files/out/");
var assemblyDirectoryPath = "/path/to/files/assemble/";
var finalizedDirectory = new java.io.File("/path/to/cron/");

var foundFiles = pollDirectory.listFiles();

if (foundFiles.length > 0) {
	// found files to batch; build file names and IDs
	var foundFileCount = 0;
	var batchFileName = "FilenameToSendToReciptient" + DateUtil.getCurrentDate('dd-MM-yy_HH-mm-ss.S');
	var outputFile = new java.io.File(assemblyDirectoryPath + batchFileName);
	var currentDateStr = DateUtil.getCurrentDate('yyyyMMddHHmmssZ');
	var controlId = DateUtil.getCurrentDate('yyyyMMddHHmmssS');

	// add batch header segments
	var headerSegments = "FHS|^~\&|SEND_APP|SEND_FACILITY|RCV_APP|RCV_FACIL|" + currentDateStr + "||||" + controlId + "\n";
	headerSegments += "BHS|^~\&|SEND_APP|SEND_FACILITY|RCV_APP|RCV_FACIL|" + currentDateStr + "||||" + controlId + "\n";
	org.apache.commons.io.FileUtils.writeStringToFile(outputFile, headerSegments, true);
	
	for each (var foundFile in foundFiles) {
		foundFileCount++;
		
		// append each found file to the output file
		org.apache.commons.io.FileUtils.writeStringToFile(outputFile, org.apache.commons.io.FileUtils.readFileToString(foundFile), true);
		
		// move each found file to processed directory
		org.apache.commons.io.FileUtils.moveFileToDirectory(foundFile, processedDirectory, true);
	}

	// add batch footer segments
	var footerSegments = "BTS|" + foundFileCount.toString() + "\n";
	footerSegments += "FTS|1\n";
	org.apache.commons.io.FileUtils.writeStringToFile(outputFile, footerSegments, true);

	// move finalized file to be picked up by scheduled SFTP sender
	org.apache.commons.io.FileUtils.moveFileToDirectory(outputFile, finalizedDirectory, true);

	var results = <batchMessage><batchCount>{foundFileCount}</batchCount><batchFileName>{batchFileName}</batchFileName></batchMessage>;

	return results;
} else {
	return null;
}
Reply With Quote
  #9  
Old 03-09-2018, 04:14 AM
odo odo is offline
OBX.3 Kenobi
 
Join Date: Feb 2017
Location: Luxembourg
Posts: 139
odo is on a distinguished road
Default

This is probably more a mid-term than a short-term solution:

I altered the Mirth server code to provide a property in the source map that indicates polling is finished when the last file is processed by a file connector.


The property is called 'pollingFinished' and is only present for the last file of a poll. There it provides the total number of polled files. It can be checked from anywhere within the channel e.g. via
Code:
if(sourceMap.containsKey('pollingFinished')){
     // do your clean-up work here
     logger.info('Polling finished. Polled ' + sourceMap.get('pollingFinished') + ' files.');
}
Please find enclosed a patch file that can be applied to the latest codebase of mirth.

Not sure if the dev-team is willing to accept community contributions (it's not even possible to upload a patch-file, thus I had to zip it...)

However it would be great to see this in the upcoming release.
Attached Images
File Type: jpg PollingFinished.jpg (38.6 KB, 74 views)
Attached Files
File Type: zip FileReceiver.java.zip (1.1 KB, 20 views)
Reply With Quote
  #10  
Old 03-09-2018, 06:40 AM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 967
agermano is on a distinguished road
Default

Maybe open a Jira ticket for a feature request and post your patch there?

http://www.mirthcorp.com/community/issues/browse/MIRTH
Reply With Quote
Reply

Tags
complete, file reader, status

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 07:38 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation