web stats
Channels not functioning after GC overhead limit reached - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 03-07-2018, 05:57 AM
BCMirthUser BCMirthUser is offline
OBX.2 Kenobi
 
Join Date: May 2015
Posts: 92
BCMirthUser is on a distinguished road
Default Channels not functioning after GC overhead limit reached

Hi All-

For reference Mirth 3.5.0.8232

I am wondering if anyone else has seen this and if so, how to fix it. We have seen scenarios in which our mirth server will run in GC overhead errors (not very often but sometimes when we implement new processes and larger than expected files get processed) and when the server restarts 1 of 3 things will happen:

1) Channel that was deployed previously will redeploy successfully and everything is fine

2) Channel redeploys, looks like it is running, but in fact is not doing anything. To fix this we have to open the channel, slightly change something (add a space to the channel name and then delete the space), save the channel, and deploy it.

3)Channel refuses to deploy. Same fix as #2 and everything is back to normal.

These seem to only pop up when we hit GC overhead errors and, while I do not have specifics, seems to effect the same channels. I dont know that this is 100% true with all the channels that were "fixed" the last time this happened.

If anyone can give me pointers on where to start investigating or if they have seen anything like this please let me know. Thank you!
Reply With Quote
  #2  
Old 03-08-2018, 02:34 AM
siddharth siddharth is online now
Mirth Guru
 
Join Date: Feb 2013
Posts: 738
siddharth is on a distinguished road
Default

I only faced GC overhead limit exceeded once when My channel was calling a recursive function, and when that happens I had to stop and start the service to bring it back to normal.

You could hit that error,when multiple channels are doing same kind of heavy work. You might want to check those heavy channels first and see if you can make them divide the efforts.
__________________
HL7v2.7 Certified Control Specialist!
Reply With Quote
  #3  
Old 03-08-2018, 05:12 AM
BCMirthUser BCMirthUser is offline
OBX.2 Kenobi
 
Join Date: May 2015
Posts: 92
BCMirthUser is on a distinguished road
Default

Siddarth-

Thanks for the reply. We are fully aware of "why" these things come up and put things into place to remedy them when they do happen. It is not often but with new processes, it may pop up.

My question more related to why we see the behavior that we do *after* this happens, and what we can do to fix it so that if we happen to run into the error it doesnt occur again.
Reply With Quote
  #4  
Old 03-08-2018, 05:28 AM
siddharth siddharth is online now
Mirth Guru
 
Join Date: Feb 2013
Posts: 738
siddharth is on a distinguished road
Default

This is just my understanding. Other explanations might differ.

Under GC overhead error your Memory might look like 7.42 GB of 7.58 usable. That's what happens - 98 percent of the memory is occupied and not getting free, so Mirth quits with that exception. While, there is ton of documentation available on GC overhead, there is hardly anything predictable about it. Mostly it is sign of broken code.
https://stackoverflow.com/questions/...limit-exceeded

Good luck.
__________________
HL7v2.7 Certified Control Specialist!

Last edited by siddharth; 03-08-2018 at 05:29 AM. Reason: foo
Reply With Quote
  #5  
Old 03-08-2018, 06:13 AM
BCMirthUser BCMirthUser is offline
OBX.2 Kenobi
 
Join Date: May 2015
Posts: 92
BCMirthUser is on a distinguished road
Default

Siddarth-

I just want to clarify, my question is not about how to fix the channel that causes the GC overhead limit, or about anything causing the GC overhead to occur. We try to avoid these at all costs but things will occasionally slip through when providing some 1 time processing of batches.

The problem is that after this happens, we restart the server, and channels, unrelated to the culprit channel that caused the issue, misbehave inconsistently. I would expect that with a full restart of the mirth server, those channels that were previously deployed and working correctly, would start back up and work as they were before the error. This does not happen. Some of the channels start back up and work just as they did, some deploy and do nothing although they say "started", some just dont deploy.

My question is about the later 2 and if there is a way we can identify why this is occurring and how we can fix it moving forward.
Reply With Quote
  #6  
Old 03-13-2018, 09:08 AM
BCMirthUser BCMirthUser is offline
OBX.2 Kenobi
 
Join Date: May 2015
Posts: 92
BCMirthUser is on a distinguished road
Default

Sorry for the Re-Bump, but I am wondering if there are any other insights on why this may happen? Just to summarize, we unexpectedly run into a GC-Overhead and are able to fix it. In the interim, restarting the Mirth Server results in 1 of 3 things happening:

1) After restart, unrelated channels deploy successfully and no other intervention is needed
2) After restart, unrelated channels deploy, say that they are "started" but are not actually functioning
3) After restart, unrelated channels will not deploy

Note that this behavior is not related to the channel that caused the GC overhead.

In the latter 2 circumstances, the fix has been to open the channel, edit the channel name by hitting the space bar, deleting the created space, saving the channel, and finally hitting deploy. We do not change any other code and the channel performs as expected after it deploys.

Do we chalk this up to unknown and unexpected behavior or is there something that could be wrong with our channel and server setup.

For reference Mirth 3.5.0.8232
Reply With Quote
  #7  
Old 03-13-2018, 11:23 AM
agermano agermano is offline
Mirth Guru
 
Join Date: Apr 2017
Location: Indiana, USA
Posts: 263
agermano is on a distinguished road
Default

That is an odd situation. I also would expect the full server restart to allow all previously working channels to continue to function as normal.

For your scenario #2, what types of channels are they? Is there anything in common with the source connector type? Have you tried actually doing an undeploy (after the server restart) before deploying them again?

For #2 or #3 are there any messages in the server log?

For #3 what happens when you try to deploy these prior to making the channel edits?
Reply With Quote
  #8  
Old 03-14-2018, 12:42 PM
BCMirthUser BCMirthUser is offline
OBX.2 Kenobi
 
Join Date: May 2015
Posts: 92
BCMirthUser is on a distinguished road
Default

Thanks for the response!

The channels are a mix of Javascript Readers, File Readers, and Channel Readers for the source.

I have not tried undeploying and then redeploying to try and fix it. I can simulate the scenario rather easily(unfortunately) so I will give this a whirl.

When I try to deploy prior to making the edits, the only thing that happens is I get pushed to the Dashboard screen as if it were a normal deploy, just no Channel shows up under its section.

The last time this happened I looked in the log and there was nothing out of the ordinary between the GC Overhead limit being reached and the restart. I will recreate the scenario and confirm that however.

I dont know if this is anything but these channels are all in Channel groups. Not in the same groups and at 4 different Channel Groups were effected the last time this happened. Within those groups, a mix of the 3 scenarios does occur.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 09:55 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Mirth Corporation