web stats
Recurring Connection Resets on HTTP Sender to specific backend. - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 09-25-2019, 01:31 PM
thaddeus_k thaddeus_k is offline
What's HL7?
 
Join Date: Sep 2019
Location: Dallas, TX
Posts: 4
thaddeus_k is on a distinguished road
Exclamation Recurring Connection Resets on HTTP Sender to specific backend.

Mirth Connect Version 3.8.0 Open Source

Issue Description:
We have a channel with an HTTP Sender in it that sends a JSON request to an external backend. When the HTTP Sender is run shortly after a previous call, (~5 minutes) everything works fine. However, if there has been a notable delay since the previous call, we instead get either a Connection Reset error, or we get a Read Timed Out error. ( The read timeout is due to an intentionally short timeout value intended to mitigate the connection reset issues).

At the moment we have mitigated this problem by implementing retry logic, but in the case we have to retry, total latency is in the realm of 8000 milliseconds. This channel services an IVR server, so expected response time is at the scale of a phone conversation with a human being on the other end of the line. An 8 second delay greatly aggravates callers.

Channel has been attached. It has two destinations, selected by the URL used by the originating requestor. (In our case, an IVR server)

A typical connection reset error as seen in mirth:
Code:
HTTP Sender error
ERROR MESSAGE: Error connecting to HTTP server
java.net.SocketException: Connection reset
<Stack trace snipped>
And a screenshot of this is attached. Notice how just ten minutes previously, a successful request with the same body went through.

Also attached are wireshark captures. Notes on those transactions follow:
Quote:
The first transaction at 8:50 AM to sales force from postman responded in 1.8 seconds and the one at 10 AM responded in 1.1 seconds. Subsequent requests on both occasions responded in less than 1 second.

The first transaction that we ran after we immediately ran the sales force transaction via mirth took us 8.6 seconds at around 8:55 am and the 2nd transaction that we ran via mirth at 10 am took 14 seconds from postman. Subsequent requests responded in less than 1 second.
Edit: the Mirth Channel is attempting to reach an HTTPS Backend , so port will be 443 if you need to filter the wireshark packets. The servers own IP address is 10.9.1.116/23, according to the Network Properties Screen.

According to nslookup from the server, target IP addresses (non-authoritative) are
Code:
nslookup <snipped>
Address:  10.9.1.230

Non-authoritative answer:
Name:    <snipped>
Addresses:  136.147.43.149
          136.147.41.149
          136.147.42.149
tl;dr This channel is giving us recurring connection resets when other channels are not, and I do not understand why it is happening or how to get it to stop.
Attached Images
File Type: png Connection_reset_error.png (38.7 KB, 5 views)

Last edited by thaddeus_k; 09-27-2019 at 07:06 AM. Reason: Add Backend URL
Reply With Quote
  #2  
Old 09-25-2019, 01:48 PM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,126
narupley is on a distinguished road
Default

Those capture files have quite a lot within them so it's hard to know what to look at. Is there a specific TCP conversation starting with a specific SYN packet or local bound ephemeral port to start with?

If the error says connection reset, that means one of the two sides is sending a RST packet to uncleanly shutdown the connection. Why that is happening can vary, though usually I've seen it happen when a data packet is sent after a connection has been closed with a FIN packet. Or in other words only one side of the conversation is cleanly shutting down.

Another possibility is the network infrastructure, like firewalls/routers in between. Is there any difference between your Postman test and the HTTP Sender test? Is that Connect server running on the same exact machine that you ran the Postman test from?
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 09-25-2019, 02:00 PM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 729
jbartels is on a distinguished road
Default

A few things to check.

First - what are the relevant IPs or ports to consider in these packet captures? They are large and unfiltered.

Next are send date and response date visible to you in MC? Turn those on as shown here: https://imgur.com/a/LEYbX3D right click on the header in the message view. What these show is when a message was sent to the destination and when the destination responded. This helps rule out problems like a slow response transformer. It tells you if the caller didn't respond fast or if your channel was slow to finish its own job.

Do you know what software or devices are between you and the receiver? Firewalls?

When you get the connection reset errors does the other applications logs show that it got and handled the request or not?
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #4  
Old 09-25-2019, 02:17 PM
thaddeus_k thaddeus_k is offline
What's HL7?
 
Join Date: Sep 2019
Location: Dallas, TX
Posts: 4
thaddeus_k is on a distinguished road
Default

Edit: the Mirth Channel is attempting to reach an HTTPS backend , so port will be 443 if you need to filter the wireshark packets.

Possibly also in the logs are the postman requests to the mirth to set the whole process off. The Mirth is listening on port 9001 for those requests.

Quote:
Is there any difference between your Postman test and the HTTP Sender test?
I am specifically remoting into the server to run my postman tests, though the recent ones don't seem to be generating the same delay.

Turning on more reporting info from the message log and attaching the screenshot. (Zipped due to size)

I am not very familiar with the network infrastructure in play, other than I know I need to use a VPN to connect to their environment.

Copying edit to original post:
Quote:
Edit: the Mirth Channel is attempting to reach an HTTPS backend , so port will be 443 if you need to filter the wireshark packets. The servers own IP address is 10.9.1.116/23, according to the Network Properties Screen.

According to nslookup from the server, target IP addresses (non-authoritative) are

Address: 10.9.1.230

Non-authoritative answer:
Name: <snipped>
Addresses: 136.147.43.149
136.147.41.149
136.147.42.149
Yet More Edits: Nick, Jon, after some more looking through things on my own, I think I have a related packet. Screenshot attached with the 'send_date' that seems to match message number 586 with packet number 815, from ip 10.9.1.116 to 136.147.41.149
It's not a SYN packet, but some sort of application data. The timestamp seems to match though, at 08:57.12.8 .
Apologies if I'm misreading, I don't have a lot of wireshark experience.
Attached Images
File Type: png message_586.png (20.7 KB, 2 views)
Attached Files
File Type: zip More_output.zip (82.5 KB, 1 views)

Last edited by thaddeus_k; 09-27-2019 at 07:07 AM.
Reply With Quote
  #5  
Old 09-26-2019, 05:22 AM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 729
jbartels is on a distinguished road
Default

I'm not quite sure whats going on here myself but I think this is one of your interactions from the "Sept 6th 10 AM" capture. https://imgur.com/a/e5i8Wqf
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #6  
Old 09-30-2019, 11:39 AM
jbartels jbartels is offline
Mirth Guru
 
Join Date: Oct 2006
Posts: 729
jbartels is on a distinguished road
Default

Thaddeus - Have you had any luck solving your issue yet?

I can try to take another look later today, but this looks tricky to diagnose.
__________________
Jon Bartels

Zen is hiring!!!!
http://consultzen.com/careers/
Talented healthcare IT professionals wanted. Engineers to sales to management.
Good benefits, great working environment, genuinely interesting work.
Reply With Quote
  #7  
Old 09-30-2019, 11:45 AM
thaddeus_k thaddeus_k is offline
What's HL7?
 
Join Date: Sep 2019
Location: Dallas, TX
Posts: 4
thaddeus_k is on a distinguished road
Default

WE have not yet solved it. I'm currently poking at the server to generate more wireshark logs (and learning more about how to filter and colorcode in WS) so that we can better characterize the events that are happening.

For one, I've noticed in the newer packet captures, that the original connection SYN-ACK isn't being included in most of my capture sessions, and that I'm instead seeing a sea of keep-alive requests before I poke the channel.

Also tried to do a redeploy with capture on to see what that looked like, but then it failed to generate the error we're trying to capture.

'll ask my managers which of these captures are clean enough to post in a public forum.
Reply With Quote
  #8  
Old 09-30-2019, 01:21 PM
thaddeus_k thaddeus_k is offline
What's HL7?
 
Join Date: Sep 2019
Location: Dallas, TX
Posts: 4
thaddeus_k is on a distinguished road
Default

Ok, I now have a fresh batch of captures from earlier today. Also included in the zip file are screenshots of wireshark as I've been looking at it, hopefully that helps guide things.

The wireshark filter I've been using is:

Code:
ip.addr eq 10.9.1.116 and ip.addr eq 136.147.40.21/16
, where 10.9.1.116 is the mirth server, and 136.147.40.21/16 seems to be the ip range my URL resolves to.

For the file named 'September_30_Two_attempts_error101538_success1016 04.pcapng', I've concluded that the two requests sent turned out to be in
Code:
tcp.stream eq 6 or tcp.stream eq 11
.

For that file, I sent one request, allowed it to reach a connection reset state, and then resent the original request, observing that it succeeded on the second try. This destination has a very long timeout to allow the full reset process to occur.


For the file named 'Sepectember_30_Prior_Auth_Timeout_Error102840_Suc cess102854.pcapng', I again sent one request on the other destination and allowed it to timeout, followed by a successful request. This one has the timeout value used in production, so that it matches what's going on in production currently. Wireshark filter is
Code:
tcp.stream eq 4 or tcp.stream eq 8

For the file named 'September_30_redeploy_and_four_successes.pcapng', I started wireshark, redeployed the mirth channel, and then sent a pair of requests on both destinations, expecting the first one of each to fail. Unexpectedly, failures did not occur on this run, so I instead have four successful processes. Here I have just been using the IP filter listed above.
Attached Files
File Type: zip Data_Collection_for_Forum.zip (5.46 MB, 1 views)
Reply With Quote
Reply

Tags
connection reset, http sender, timeout

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 01:04 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Mirth Corporation