web stats
File Reader - Ignore Carriage Returns - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 03-27-2013, 04:19 AM
jacob41 jacob41 is offline
Mirth Newb
 
Join Date: Jan 2013
Posts: 19
jacob41 is on a distinguished road
Default File Reader - Ignore Carriage Returns

Prereqs:
What version of Mirth are you using?
2.2.1

Post your channel.
attached

Tell us about what systems you're communicating with.
None - this channel just translates a .csv to xml. Later, I load results to a SQL Server db using a web service.

Post any error messages.
No errors, but blank lines (CR LF) are read and converted to XML messages.

Log on to IRC.
on as jacob41.

Let us know what solved the problem.
n/a.

Issue:
File reader reads blank lines and converts them to non-meaningful messages.

Details
Summary > Set Data Types > Properties > Ignore Carriage Returns is checked.

The data file is actually a csv, but uploads of this type are not allowed on your site.

I don't want to have to preprocess csv files to remove blank (actually 0xD 0xA) lines. I want an automated process for any number of blank lines.

Processing the attached example with this channel, I would get one meaningful record and approximately 5 nearly blank records added to my db.

Examples - Source/Destination
Source
<delimited>
<row>
<column1/>
</row>
</delimited>
Destination
<delimited>
<row>
<PracticeCode>DEFAULT</PracticeCode>
<MemberID/>
<MemberStatus>ADD</MemberStatus>
<FirstName/>
<MiddleName/>
<LastName/>
<AddressLine1/>
<AddressLine2/>
<City/>
<State/>
<Zip/>
<DOB/>
<Gender/>
<SSN/>
<HomePhone/>
<WorkPhone/>
<CellPhone/>
<ProgramCode>DEFAULT</ProgramCode>
<ProviderCode>DEFAULT</ProviderCode>
</row>
</delimited>
Attached Files
File Type: xml CMP sftp.xml (38.8 KB, 21 views)
File Type: txt sampleData_CMP.txt (248 Bytes, 40 views)

Last edited by jacob41; 03-27-2013 at 04:21 AM. Reason: I left out a general description of my results.
Reply With Quote
  #2  
Old 03-27-2013, 06:16 AM
cory_cole cory_cole is offline
Mirth Guru
 
Join Date: Mar 2012
Posts: 1,282
cory_cole is on a distinguished road
Default

You can do this in the preprocessor and do global search and replace. That will handle any number of blank lines.
Reply With Quote
  #3  
Old 03-27-2013, 09:17 AM
jacob41 jacob41 is offline
Mirth Newb
 
Join Date: Jan 2013
Posts: 19
jacob41 is on a distinguished road
Default

What do you mean by 'preprocessor'?

I suppose you're referring to pre-XML, and so I'm looking at the Source tab. I don't see anything that could perform the functionality you mention.

Thank you.
Reply With Quote
  #4  
Old 03-27-2013, 09:24 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,123
narupley is on a distinguished road
Default

Unfortunately a preprocessor won't quite work since you're trying to handle batching at the same time. A batch script will work though:

Source Inbound Delimited Text Properties
  • Column Names: Location,MRN,Gender,DOB,Age,FirstName,MiddleName,L astName,Addr1,City,State,Zip,HomePhone,CellPhone
  • Ignore Carriage Returns: No
  • Number of Header Records: 1
  • Split Batch by Record: No
  • Split Batch by Javascript:
    Code:
    var line;
    while ((line = reader.readLine()) != null)
    	if (!line.isEmpty())
    		return line;
    return '';

Make sure you copy the batch script over to the source outbound data type as well, due to MIRTH-2210 (which will be fixed in the 3.0 release).

The above assumes that your header row will always be static (you hard-code the column names). If instead you want the column names to be dynamic depending on the first row, you can do this instead:

Source Inbound Delimited Text Properties
  • Column Names: <blank>
  • Ignore Carriage Returns: No
  • Number of Header Records: <blank, or zero>
  • Split Batch by Record: No
  • Split Batch by Javascript:
    Code:
    var line;
    while ((line = reader.readLine()) != null) {
    	if (!line.isEmpty()) {
    		if (!$gc('header'))
    			$gc('header',line.replaceAll('\\s','').split(','));
    		else {
    			var msg = <delimited><row/></delimited>;
    			var columns = line.split(',');
    			for (var i = 0; i <= columns.length-1; i++)
    				msg.row[$gc('header')[i]] = columns[i];
    			return msg;
    		}
    	}
    }
    globalChannelMap.remove('header');
    return '';

Source Transformer
Code:
msg = new XML(messageObject.getRawData());
It also seems like you're doing a lot of superfluous Mapper / Message Builder steps that can be much more simply handled in a single transformer... I'll leave that to you as to whether you want to clean it up though.
Attached Files
File Type: xml CMP sftp Modified (Static Columns).xml (39.2 KB, 34 views)
File Type: xml CMP sftp Modified (Dynamic Columns).xml (40.7 KB, 22 views)
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #5  
Old 03-27-2013, 11:34 AM
jacob41 jacob41 is offline
Mirth Newb
 
Join Date: Jan 2013
Posts: 19
jacob41 is on a distinguished road
Default

Thank you for your reply - and our discussion via IRC.

However, I imported your first example and it does not work. (Edit: This morning, I imported your second example - it suffers from the same problem.)

Now, instead of only empty lines becoming empty records, every record is emptied. See:

<delimited>
<row>
<PracticeCode>DEFAULT</PracticeCode>
<MemberID/>
<MemberStatus>ADD</MemberStatus>
<FirstName/>
<MiddleName/>
<LastName/>
<AddressLine1/>
<AddressLine2/>
<City/>
<State/>
<Zip/>
<DOB/>
<Gender/>
<SSN/>
<HomePhone/>
<WorkPhone/>
<CellPhone/>
<ProgramCode>DEFAULT</ProgramCode>
<ProviderCode>DEFAULT</ProviderCode>
</row>
</delimited>

Last edited by jacob41; 03-28-2013 at 04:22 AM. Reason: Tried second proposed solution.
Reply With Quote
  #6  
Old 08-12-2013, 08:34 AM
rav rav is offline
Mirth Guru
 
Join Date: Oct 2012
Posts: 258
rav is on a distinguished road
Default

Did you get this resolved..Can you please guide me.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 01:16 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation