web stats
HL7 Acknowledgement packet not sent properly over tcp - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 02-22-2018, 02:39 PM
greentea greentea is offline
What's HL7?
 
Join Date: Feb 2018
Posts: 2
greentea is on a distinguished road
Question HL7 Acknowledgement packet not sent properly over tcp

Hi there,

I have a tcp listener setup on Linux with Mirth with the following config:

Quote:
Listener settings
- All interfaces

Source settings
- Source Queue: OFF (Respond after processing)
- Response: Auto-generate (After source transformer)
- Process Batch: No
- Max processing threads: 1

TCP Listener Settings
- Tx Mode: MLLP
- Mode: Server
- Max Connections: 10
- Receive Timeout (ms): 0
- Buffer Size (bytes): 65536
- Keep Connection Open: Yes
- Data Type: Text
- Encoding: Default
- Respond on New Connection: No
Our client sends us ADT messages, and we are able to receive them. ACK messages are being sent from Mirth correctly as well, however, only some make it to the client. Hence, the client's application treats the message as 'timed out' and puts it in an error queue and re-tries several times.

Our tcpdump shows that Mirth indeed sends out the ACKs but client's side sees some of the ACKs and some others are broken (i.e., their received packet length does not match our sent packet length). This causes multiple re-transmission attempts from our side, to no avail.
Frustratingly, this does not happen all the time -- more like 3/10 times. It seems to happen more often when the client sends us multiple (3~5) ADT messages in rapid succession (They have a DB poller on their side that gathers up all ADT messages and sends them out all at once).

We initially thought it was an issue with the firewall, but we both have all ports open and some of our ACKs DO make it to our client.

I have also tested with HAPI on my local environment, with exactly the same message contents that failed previously, and was able to send in rapid succession with no packet loss.

If you could provide any insight, I would greatly appreciate it. Please let me know if you need any other information.

PS: Is there a technical support personnel I can contact directly with Mirth? My company is willing to pay for professional support.

Thank you.
Reply With Quote
  #2  
Old 02-22-2018, 02:59 PM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,116
narupley is on a distinguished road
Default

That is indeed odd. Try taking network captures on both the client side and the Mirth Connect side. Compare the packets for each. Do they have the same TCP sequence numbers?

I think the troubleshooting you've already done more or less proves that something in the middle, in between the client and server, is messing up the packets?
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.

Last edited by narupley; 02-22-2018 at 03:22 PM.
Reply With Quote
  #3  
Old 02-23-2018, 08:52 AM
greentea greentea is offline
What's HL7?
 
Join Date: Feb 2018
Posts: 2
greentea is on a distinguished road
Question

Quote:
Originally Posted by narupley View Post
Try taking network captures on both the client side and the Mirth Connect side. Compare the packets for each. Do they have the same TCP sequence numbers?
Here is the comparison (LINK):


Our side's IP is 10.... and remote side is 192...
In this scenario, Remote Side is sending us an ADT message and we are responding back with ACK.
Looking at the remote side, packets from No. 1 to 9 are from one attempt, and 10 to 19 are from second attempt, about 5 minutes later.

From the first attempt, we see that two ADT messages (top two green lines) are received on our side, and two ACK messages are sent. These packets get messed up (indicated by red line) and only one is received, and its Seq and Ack do not match -- I'm not sure what this means.

Then, I am not sure what TCP Retransmission (No.4 to 8) is trying to do. Is the remote side detecting that the packet is messed up and is requesting our side to send it again?

On our side, at No. 215, we also have a bunch of TCP Retransmission, for the second ack message it seems.

During the second attempt, we see the two ADT messages get received and ACK sent back correctly.

Is there any configuration on Mirth's side that I could change to improve the stability? Would setting the Keep Connection Open under TCP Listener setting to FALSE help?


Thank you very much.
Reply With Quote
  #4  
Old 02-23-2018, 10:28 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,116
narupley is on a distinguished road
Default

Not 100%, but I think the 4-8 on the remote side may indicate a completely new message that the remote side is trying to send to Mirth, but that too is failing. So the retransmissions are not just from the Mirth side to remote, but both ways, and happening at the same time. This may be more evidence of something external (firewall, router, etc.) causing these networking issues.

Turning Keep Connection Open off may help because that will force the client to open a completely new connection for each message. You'll still likely see some errors on the client side because it'll probably still attempt to send multiple messages in bursts over the same connection. On the Mirth Connect side the connection will be closed after the first message, regardless of whether the socket buffer contains more data. And also, if something external is causing these networking issues, you'll still see the same errors and symptoms (though perhaps slightly less often).

Another thing that may help is to turn the Source Queue on, and set the response to Auto-Generate. Then the channel will respond with an ACK immediately after receiving the message (and committing the raw data to the database). The message is then processed through the channel asynchronously. The caveat to this method is that if you're building up a custom response ACK to send back, you'll no longer be able to do that with the source queue on.

And again even with the source queue option, it won't prevent those networking issues (as they are likely caused by something external). But it may mitigate the effects of them.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
Reply

Tags
acknoweldgement, adt, hl7, tcp

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 03:41 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation