web stats
Mirth Connect behind a load balancer - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 10-08-2018, 06:32 AM
babston babston is offline
What's HL7?
 
Join Date: Oct 2018
Posts: 4
babston is on a distinguished road
Default Mirth Connect behind a load balancer

We have two virtual servers running Mirth Connect on each. The load balancer is setup to pass traffic to the server over certain ports that the Mirth interfaces are listening on.

The question I have is that since we put these two servers behind the load balancer the max connections seem to be going up and maxing out at 20 (per our setting) where this never happened before they were load balanced. Do you know what would be making this happen?

Our setting are below:
Windows 2016
Connector type is TCP Listener
MLLP
Max Connections 20
Receive Timeout 0
buffer size 65536
Keep connection open Yes


We are also using this error.. I have not been able to find much on it and maybe it could be causing our issues as well:


[2018-10-08 09:20:40,340] ERROR (com.mirth.connect.connectors.tcp.TcpReceiver:740) : Error receiving message (TCP Listener "Source" on channel 7530dfb7-11f4-401f-b56d-f8e9c6e1f487).
java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at com.mirth.connect.model.transmission.framemode.Fra meStreamHandler.read(FrameStreamHandler.java:120)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.readBytes(TcpReceiver.java:807)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:614)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:506)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker( Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)
Reply With Quote
  #2  
Old 10-08-2018, 07:08 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,050
narupley is on a distinguished road
Default

It sounds like the load balancer software you're using is opening connections to Connect and not closing them.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 10-08-2018, 07:36 AM
babston babston is offline
What's HL7?
 
Join Date: Oct 2018
Posts: 4
babston is on a distinguished road
Default

Quote:
Originally Posted by narupley View Post
It sounds like the load balancer software you're using is opening connections to Connect and not closing them.
ok. I will pass along to see if the LB is doing that. Any thoughts on the error below. It has started with us moving the servers behind the load balancer as well.

[2018-10-08 09:20:40,340] ERROR (com.mirth.connect.connectors.tcp.TcpReceiver:740) : Error receiving message (TCP Listener "Source" on channel 7530dfb7-11f4-401f-b56d-f8e9c6e1f487).
java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at com.mirth.connect.model.transmission.framemode.Fra meStreamHandler.read(FrameStreamHandler.java:120)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.readBytes(TcpReceiver.java:807)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:614)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:506)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker( Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)
Reply With Quote
  #4  
Old 10-09-2018, 05:33 AM
arun.sutharshan arun.sutharshan is offline
Mirth Newb
 
Join Date: Apr 2014
Location: London, United Kingdom
Posts: 9
arun.sutharshan is on a distinguished road
Default

Quote:
Originally Posted by babston View Post
We have two virtual servers running Mirth Connect on each. The load balancer is setup to pass traffic to the server over certain ports that the Mirth interfaces are listening on.

The question I have is that since we put these two servers behind the load balancer the max connections seem to be going up and maxing out at 20 (per our setting) where this never happened before they were load balanced. Do you know what would be making this happen?

Our setting are below:
Windows 2016
Connector type is TCP Listener
MLLP
Max Connections 20
Receive Timeout 0
buffer size 65536
Keep connection open Yes


We are also using this error.. I have not been able to find much on it and maybe it could be causing our issues as well:


[2018-10-08 09:20:40,340] ERROR (com.mirth.connect.connectors.tcp.TcpReceiver:740) : Error receiving message (TCP Listener "Source" on channel 7530dfb7-11f4-401f-b56d-f8e9c6e1f487).
java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at com.mirth.connect.model.transmission.framemode.Fra meStreamHandler.read(FrameStreamHandler.java:120)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.readBytes(TcpReceiver.java:807)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:614)
at com.mirth.connect.connectors.tcp.TcpReceiver$TcpRe ader.call(TcpReceiver.java:506)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker( Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)

Are these two mirth instances are managed via a mirth appliance

If not, how does the current LB manage delivering message to a specific Mirth instance (and returning ACK back if applicable)?

In general, only one Mirth instance should be active at a time.
Reply With Quote
  #5  
Old 10-10-2018, 02:33 AM
babston babston is offline
What's HL7?
 
Join Date: Oct 2018
Posts: 4
babston is on a distinguished road
Default

Quote:
Originally Posted by arun.sutharshan View Post
Are these two mirth instances are managed via a mirth appliance

If not, how does the current LB manage delivering message to a specific Mirth instance (and returning ACK back if applicable)?

In general, only one Mirth instance should be active at a time.
These are two windows VMs tuning Mirth Connect. No Mirth appliance. They are behind a Foritnet ADC. The load balancer knows the source IPs and port and routes to the two server via set ports per interface on least number of connections. The VMs send back the ack directly to the sending system. I am thinking these errors are from the load balancer health checks but not 100% sure. We just moved to this setup and started seeing these errors.
Reply With Quote
  #6  
Old 10-10-2018, 02:35 AM
babston babston is offline
What's HL7?
 
Join Date: Oct 2018
Posts: 4
babston is on a distinguished road
Default

Quote:
Originally Posted by narupley View Post
It sounds like the load balancer software you're using is opening connections to Connect and not closing them.
So looks like the LB had a very low timeout and was holding those connections open which was causing them them hit the max connection limit. We raised the timeout to 300 seconds and the connection count issues has gone away.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 01:24 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Mirth Corporation