web stats
Mirth Connect 3.0: Deadlocks - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 11-03-2013, 05:06 PM
jonlent jonlent is offline
Mirth Newb
 
Join Date: Aug 2011
Location: Dallas, Texas, United States of America
Posts: 17
jonlent is on a distinguished road
Unhappy Mirth Connect 3.0: Deadlocks

We have numerous inbound channels that are MLLP listeners sending to multiple channel-reader type destinations. Everything is queued and none of it is synchronized.

Frequently, this configuration produces a deadlock while updating the message statistics for the channel and destinations:

Code:
[2013-11-03 18:47:36,366]  ERROR (com.mirth.connect.connectors.vm.VmDispatcher:510): Error processing queued message 215-3 (SENT) for channel 8c89cb69-b26d-450e-a9fc-d09207d9b6eb (Destination 3). This error is expected if the message was manually removed from the queue.
com.mirth.connect.donkey.server.data.DonkeyDaoException: java.sql.SQLException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.addChannelStatistics(JdbcDao.java:369)
	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.commit(JdbcDao.java:1581)
	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.executeTasks(BufferedDao.java:131)
	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:74)
	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:61)
	at com.mirth.connect.donkey.server.channel.DestinationConnector.run(DestinationConnector.java:498)
	at java.lang.Thread.run(Unknown Source)Caused by: java.sql.SQLException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
	at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2894)
	at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2334)
	at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:643)
	at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:614)
	at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:573)
	at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:707)
	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.addChannelStatistics(JdbcDao.java:346)
	... 6 more
Turning the synchronization (wait for previous channel...) on makes this issue go away, but makes our processing sequential -- which we would like to avoid.

I've attached the offending configuration, and I can get it to reliably trigger using a set of about 500 messages pumped at it.

I've started to look at the way the code for the channels queues up the events for processing based on the Deadlock Graph included in the ZIP file -- but that is as far as I have gotten thus far.

Just wanted to see if anyone else had seen this and if anybody was out ahead of me on the analysis.

Oh -- the messages do get delivered, it is just the stat updates that are failing.

Environment:
  • Windows Server 2008 R2
  • Mirth Connect Server 3.0.0.6931
  • Built on September 30, 2013
  • Server ID: 2f1516e3-8ed6-443e-914f-c670e8fde5cf
  • Java version: 1.7.0_45 (64-bit)
  • SQL Server 2008 R2 & SQL Server 2012

Cordially,

Jonathan Lent
Attached Files
File Type: zip Deadlock Sandbox.zip (12.6 KB, 3 views)
Reply With Quote
  #2  
Old 11-04-2013, 08:21 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

Thank you for finding this, I've opened an issue for it: MIRTH-3042
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 05-14-2015, 05:51 AM
csummer csummer is offline
Mirth Newb
 
Join Date: Oct 2013
Posts: 8
csummer is on a distinguished road
Default So what is the resolution to the Deadlock in SQL?

Reply With Quote
  #4  
Old 05-14-2015, 07:46 AM
narupley's Avatar
narupley narupley is online now
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

Quote:
Originally Posted by csummer View Post
The issue has been fixed, so the resolution is to upgrade your Mirth Connect instance to the latest.
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
Reply

Tags
3.0, connect, deadlock, sql server

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 12:37 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation