web stats
How rollback Mirth connect cluster - Page 2 - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #11  
Old 10-29-2015, 04:40 AM
cguanes cguanes is offline
OBX.2 Kenobi
 
Join Date: Jan 2011
Posts: 55
cguanes is on a distinguished road
Thumbs down How rollback Mirth connect cluster - Crashing

You are welcome, I hope people read this thread before doing the same mistake that I did.

I have a PICO for development so I couldn't try the cluster first (I just test channels code compatibility), and also the cluster ran for two days so I didn't care to move my snapshot to a safe place...

And don't trust that mirth support will get you ASAP... they don't care much about your emergency or your patients
Reply With Quote
  #12  
Old 02-01-2016, 09:35 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 565
kirbykn2 is on a distinguished road
Default Advanced Clustering update

Hello Carlos,

Do you know if Mirth has resolved the issues with advanced clustering yet?
__________________
Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.2
Mirth Connect Version 3.6.1
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_121-b13
PostgreSQL Version 9.6.3
Reply With Quote
  #13  
Old 02-01-2016, 09:37 AM
cguanes cguanes is offline
OBX.2 Kenobi
 
Join Date: Jan 2011
Posts: 55
cguanes is on a distinguished road
Default

Quote:
Originally Posted by kirbykn2 View Post
Hello Carlos,

Do you know if Mirth has resolved the issues with advanced clustering yet?
I don't know, I didn't try it anymore... Let see if they answer here..
Reply With Quote
  #14  
Old 02-01-2016, 11:34 AM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

First of all we appreciate your patience as we work on ironing out the issues with database replication + clustering.

Here is a status update based on the main issues I was able to gather from reading this thread:

1) Outbound traffic not coming from the cluster VIP.
2) Cluster crashing unexpectedly.
3) Problems configuring.

Comments on each:

1) Outbound traffic not coming from the cluster VIP.

We are aware that this is a showstopper for those wanting upgrade to the new db replication-based clustering. It is on the Mirth Appliance roadmap to bring back this feature from the old clustering service.

2) Cluster crashing unexpectedly.

To be able to narrow this down to a specific cause, we need more information about your specific environment, what the servers were doing at the time of the crash and what was shown in the mirth connect server logs at the time (preferably with debug logging turned on).

That being said, I suspect that an unexpected shutdown of the whole cluster (due to all servers heartbeats' timing out) could occur if the shared database is inaccessible for longer than the heartbeat timeout period (default 5 mins). The auto-shutdown on heartbeat timeout is intended to protect against concurrent access to shared resources, which is a new danger presented by the shared / replicated database architecture. However if the database itself stalls, the whole cluster shouldn't have to halt. This is also on our roadmap to address in an upcoming version of the Mirth Connect clustering plugin and is actively being worked on.

3) Problems configuring.

After the initial release of database replication + clustering, many issues related to installation/setup on Mirth Appliances were brought to our attention. For those issues that we have documented, some have already been fixed and rolled out via Mirth Appliance updates and the remainder will be fixed in subsequent updates.

If there are other issues or concerns, please let us know as we'd like to make sure all the problems in the initial rollout have been documented and added to our development roadmaps. And please remember that providing as much detail as possible is critical to help us determine the root cause of the issue! Thanks.
Reply With Quote
  #15  
Old 02-02-2016, 04:32 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 565
kirbykn2 is on a distinguished road
Default Date

Thanks for the detailed explanation.

Do you have any idea when this fixes will be in place?
__________________
Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.2
Mirth Connect Version 3.6.1
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_121-b13
PostgreSQL Version 9.6.3
Reply With Quote
  #16  
Old 02-02-2016, 10:17 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 565
kirbykn2 is on a distinguished road
Default Jira

Or better yet, are these issues entered in Jira? If so, please provide the issue numbers and I will track there. If not, can you have them entered?
__________________
Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.2
Mirth Connect Version 3.6.1
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_121-b13
PostgreSQL Version 9.6.3
Reply With Quote
  #17  
Old 02-03-2016, 02:05 PM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

Quote:
Do you have any idea when this fixes will be in place?
The most I can say at this point is that some fixes have already been made available in Mirth Appliance updates, some are scheduled for release with Mirth Connect version 3.4.0 (eta early March) and the rest hopefully soon after that.

Quote:
Or better yet, are these issues entered in Jira?
Only issues related to the open-source distribution are tracked in the public JIRA project. Since these issues fall under Mirth Appliance and the Mirth Connect commercial plugins (which are not open source) they are not tracked in the public JIRA.

Quote:
Mirth is not presenting the Cluster VIP address as the source IP.
Are your destination connectors TCP Senders? If so have you tried entering the cluster VIP address in the "Override Local Binding" option?

Also, is there a particular reason the downstream systems can't accept connections from the individual IP addresses of the cluster? We may be able to suggest some alternate solutions for you.

Thanks again.
Reply With Quote
  #18  
Old 02-05-2016, 04:59 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 565
kirbykn2 is on a distinguished road
Default

There are multiple reasons why downstream systems are unable to accept connections from multiple IP's, interface is configured to keep the connection open once its established, for security reasons the interface is configured to only accept connections from one IP, once a connection is established the only way to receive a new connection is to restart the interface.

Can you send me more information and what exactly was fixed with advanced clustering and replication?
__________________
Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.2
Mirth Connect Version 3.6.1
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_121-b13
PostgreSQL Version 9.6.3
Reply With Quote
  #19  
Old 02-05-2016, 03:12 PM
brentm brentm is offline
Mirth Employee
 
Join Date: Jan 2012
Posts: 85
brentm is on a distinguished road
Default

Quote:
Can you send me more information and what exactly was fixed with advanced clustering and replication?
The fixes that have been released so far were related to database replication and were included in the Postgres 9.4.5 update for Mirth Appliances. These may address some of the configuration problems users have experienced. You can request a detailed changelog from Mirth Support if you'd like a full list of the changes and improvements.

Additional fixes related to the clustering setup wizard on Mirth Appliances and an issue related to the cluster crashing are tentatively scheduled to be released with Mirth Connect 3.4.0.

Quote:
There are multiple reasons why downstream systems are unable to accept connections from multiple IP's, interface is configured to keep the connection open once its established, for security reasons the interface is configured to only accept connections from one IP, once a connection is established the only way to receive a new connection is to restart the interface.
Regarding the outbound VIP binding issue, is your cluster an active/passive (failover) or active/active (load-balanced) cluster?

Also, are you using TCP Senders and if so have you tried using the "Override Local Binding" option?
Reply With Quote
  #20  
Old 02-09-2016, 05:13 AM
kirbykn2's Avatar
kirbykn2 kirbykn2 is offline
Mirth Guru
 
Join Date: Sep 2014
Location: Michigan
Posts: 565
kirbykn2 is on a distinguished road
Default

We are currently using the active/passive mode. We are not using "Override Local Binding".

Perhaps you can review these support tickets, and let me know if all of these issues that were discovered during our installation have been resolved.

[ 37865 ] Munson - Uninstalling the MC clustering plugin clustering causes loss of channel revisions SUPPORT-3637

[ 36951 ] Munson - Installing & setting up Advanced Clustering
__________________
Kirby

Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

Appliance Version 3.11.2
Mirth Connect Version 3.6.1
Java Version 1.6.0_45-b06
Java (64 bit) Version 1.6.0_45-b06
Java 7 (64 bit) Version 1.7.0_151-b15
Java 8 (64 bit) Version 1.8.0_121-b13
PostgreSQL Version 9.6.3
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 02:15 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation