web stats
Clearing out old socket connections - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 05-09-2014, 01:37 PM
marke marke is offline
Mirth Newb
 
Join Date: Jan 2013
Posts: 20
marke is on a distinguished road
Talking Clearing out old socket connections

Scenario:
A live connection exists between Mirth (using TCP Listener) and the Client. The Client, (for whatever reason), decides to close the connection and open a new connection to send messages.

Perception:
We've set the [Max Connection] setting = 1 because we do not want the Client sending messages to the channel thru more than one connection simultaneously. Now, because of this setting, it appears that Mirth will not see the new connection established by the client unless the Mirth channel is redeployed. So anytime the Client closes the connection and opens a new connection, it appears that Mirth is still looking at the old connection that was originally established with the Client.

Question:
If the Max Connection setting = 1, is there a way that Mirth can detect if the Client has closed its existing connection - and then automatically re-connect
to the new active Client connection without having to re-deploy the channel?

Reply With Quote
  #2  
Old 05-09-2014, 01:44 PM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

If the client closes its connection, then the server (TCP Listener) will detect that and shutdown the local socket as well, so that another client connection can connect. I'm not able to reproduce the scenario you described. Can you do a network capture to see exactly what's going on?
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 05-12-2014, 06:55 AM
StefanScholte StefanScholte is offline
 
Join Date: May 2009
Location: Netherlands, Harderwijk
Posts: 321
StefanScholte is on a distinguished road
Default

I have something similar but I'm not able to reproduce this. but I have channel with 4 connections connected (this is wat Mirth Shows) but there is no client connected at this moment.

only a restart of the mirth Service will close the connections.
Reply With Quote
  #4  
Old 05-12-2014, 01:13 PM
DannyCortes DannyCortes is offline
What's HL7?
 
Join Date: Feb 2014
Posts: 4
DannyCortes is on a distinguished road
Default

Hi narupley,

Here is my theory, I believe the issue is the Client not closing the connection when is done sending a message.

Possible flow:
Client sends a message and does not close the socket, Mirth waits for the timeout and then closes it. The client then tries to send a message using the same socket, but this time it is closed, so it opens a new connection and starts sending out messages to that new socket but it encounters an issue because mirth is configured to have a maximum of 1 connections ( we want to maintain the order of precedence) and in theory there's another connection already stablished with them.

We would like to have some guidance on this matter because the only way we know the channel is not listening is when the client makes us aware of the issue. The way we momentarily fix the issue is by manually redeploying the affected channels.

I am pretty sure this is related to the same issue but we get a lot of connection reset errors (Error Message = Connection reset Error Type = Source Connector)

Last edited by DannyCortes; 05-12-2014 at 01:15 PM.
Reply With Quote
  #5  
Old 05-12-2014, 01:21 PM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

Yes, that theory is sound. It's not a problem with Mirth Connect, it's a problem with the connecting client. If you configure your server to have a maximum of X connected sockets and the client isn't closing sockets appropriately, then the maximum will be reached and there's nothing the server can do, because it has no idea whether the client is actually "done with" the socket or not.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #6  
Old 05-16-2014, 04:44 PM
DannyCortes DannyCortes is offline
What's HL7?
 
Join Date: Feb 2014
Posts: 4
DannyCortes is on a distinguished road
Default

After some research and with the help of Wireshark, I was able to find the exact moment when Mirth stops ingesting messages but I don't know how to interpret this information and why this is happening.
The last received message and normal communication that we had with the Client was at 5:23 AM then at 5:41 the client changed the sending port and Mirth stopped ingesting messages, then around 7:23 We got a connection reset email from Mirth and we started ingesting messages again.

Port 7154 is the Mirth Channel Port (Receiver)
Port 60843 is the Client Port (Sender)
Port 52425 is another Client Port (Sender)

I couldn't upload the screen shot but here is the chain of events.
Can you please point out what's going on here?

5:23 AM 60843 > 7154 [PSH, ACK] Seq=14746495 Ack=107641 Win=130048 Len=173
5:23 AM 7154 > 60843 [ACK] Seq=107641 Ack=14746668 Win=211712 Len=0
5:23 AM 7154 > 60843 [PSH, ACK] Seq=107641 Ack=14746668 Win=211712 Len=78
5:23 AM 60843 > 7154 [ACK] Seq=14746668 Ack=107719 Win=130048 Len=0
5:41 AM 52425 > 7154 [SYN] Seq=0 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
5:41 AM 7154 > 52425 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
5:41 AM 52425 > 7154 [ACK] Seq=1 Ack=1 Win=131072 Len=0
5:41 AM 7154 > 52425 [FIN, ACK] Seq=1 Ack=1 Win=131072 Len=0
5:41 AM 52425 > 7154 [PSH, ACK] Seq=1 Ack=1 Win=131072 Len=200
5:41 AM 7154 > 52425 [RST, ACK] Seq=2 Ack=201 Win=0 Len=0
5:41 AM 52425 > 7154 [PSH, ACK] Seq=201 Ack=1 Win=131072 Len=200
5:41 AM 7154 > 52425 [RST] Seq=1 Win=0 Len=0 |After this Mirth stopped ingesting messages and we started sending RST back to the client and not ACK’s but the client kept sending us messages.
5:41 AM - 7:23 AM A whole bunch of [PSH,ACK] From the client and [RST] From Mirth, A couple of [SYN] From the Client and [SYN,ACK] From Mirth and Two Retransmitions
7:23 AM [TCP Keep-Alive] 7154 > 60843 [ACK] Seq=107718 Ack=14746668 Win=211712 Len=1 | We had 10 of this back to back
7:23 AM 7154 > 60843 [RST, ACK] Seq=107719 Ack=14746668 Win=0 Len=0 |After this We went back to normal and got flooded with messages.
7:23 AM 53978 > 7154 [SYN] Seq=0 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
7:23 AM 7154 > 53978 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
7:23 AM 53978 > 7154 [ACK] Seq=1 Ack=1 Win=131072 Len=0
7:23 AM 53978 > 7154 [PSH, ACK] Seq=1 Ack=1 Win=131072 Len=200
Reply With Quote
  #7  
Old 05-17-2014, 08:59 AM
DannyCortes DannyCortes is offline
What's HL7?
 
Join Date: Feb 2014
Posts: 4
DannyCortes is on a distinguished road
Default

I forgot to mention that we changed the timeout from 10 seconds to 0 (infinite) and we kept the maximum connections set to 1 and keep connection open set to yes
Reply With Quote
  #8  
Old 05-19-2014, 06:38 AM
marke marke is offline
Mirth Newb
 
Join Date: Jan 2013
Posts: 20
marke is on a distinguished road
Default

I wonder if the client is changing the connection but not closing the original connection, so Mirth is sending RST over and over:
7:23 AM 7154 > 60843 [RST, ACK] Seq=107719 Ack=14746668 Win=0 Len=0
to tell the Client to close the original connection - while they are sending messages over the new connection?
Reply With Quote
  #9  
Old 05-19-2014, 06:54 AM
DannyCortes DannyCortes is offline
What's HL7?
 
Join Date: Feb 2014
Posts: 4
DannyCortes is on a distinguished road
Default

What about this
7:23 AM [TCP Keep-Alive] 7154 > 60843 [ACK] Seq=107718 Ack=14746668 Win=211712 Len=1

We had 10 of those back to back and they are directed to the port they stopped using at 5:23 AM. I am wondering why this is being sent two hours later.
Reply With Quote
  #10  
Old 05-19-2014, 08:17 AM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

The network capture you posted confirms my suspicions that the client is simply not closing its socket like it should be. Therefore since you have a maximum of 1 connection set on the TCP Listener, what you're seeing is the expected behavior.

Client 1 (60843) sends a message and gets an ACK, but never sends a FIN packet. When Client 2 (52425) connects, the server sees that the maximum number of connected sockets (1) has already been reached, so the server-side socket gets closed (you see the FIN,ACK sent back to Client 2). Client 2 ignores the FIN packet and tries to send data, and the server obviously returns a RST because that connection is no longer valid.

Let me reiterate that this is what should happen when you have "Max Connections: 1" set. The problem lies not with Mirth Connect but instead with the connecting client that is not closing its socket when it is done sending a message. The Keep-Alive ACKs you get further prove that the network layer on the remote client side still thinks that the connection is valid. You need to either increase the maximum number of connections on the TCP Listener, or have the remote side fix its application to close connections correctly.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
Reply

Tags
tcp listener connection

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 11:11 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation