web stats
Error on reattachment - deadlock SQL - Mirth Community

Go Back   Mirth Community > Mirth Connect > Support

Reply
 
Thread Tools Display Modes
  #1  
Old 10-08-2019, 08:08 AM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default Error on reattachment - deadlock SQL

Hi,

I'm having problems when trying to process DICOM images with several processing threads.
They are received by 1 channel (with queue for speed), and split to several channels to be send out to one 'To PACS' channel. But i get this error on all channels except for the receiving one (has around 100 incoming messages a second).

Can i do something about this, or is the only way to fix this to set the processing threads to 1 for all channels?

Mirth Connect Server 3.8.0
Microsoft SQL Server 2017
Java version: 1.8.0_181

Code:
[2019-10-08 05:59:51,669]  ERROR  (com.mirth.connect.server.attachments.dicom.DICOMAttachmentHandlerProvider:370): Error reattaching attachments
com.mirth.connect.donkey.server.data.DonkeyDaoException: java.sql.SQLException: Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.getMessageAttachment(JdbcDao.java:1529)
	at com.mirth.connect.server.controllers.DonkeyMessageController.getMessageAttachment(DonkeyMessageController.java:283)
	at com.mirth.connect.server.util.DICOMMessageUtil.getDICOMRawBytes(DICOMMessageUtil.java:85)
	at com.mirth.connect.server.attachments.MirthAttachmentHandlerProvider.reAttachMessage(MirthAttachmentHandlerProvider.java:132)
	at com.mirth.connect.server.attachments.MirthAttachmentHandlerProvider.reAttachMessage(MirthAttachmentHandlerProvider.java:60)
	at com.mirth.connect.server.attachments.MirthAttachmentHandlerProvider.reAttachMessage(MirthAttachmentHandlerProvider.java:56)
	at com.mirth.connect.connectors.vm.VmDispatcher.send(VmDispatcher.java:108)
	at com.mirth.connect.donkey.server.channel.DestinationConnector.handleSend(DestinationConnector.java:888)
	at com.mirth.connect.donkey.server.channel.DestinationConnector.process(DestinationConnector.java:518)
	at com.mirth.connect.donkey.server.channel.DestinationChain.doCall(DestinationChain.java:121)
	at com.mirth.connect.donkey.server.channel.DestinationChain.call(DestinationChain.java:63)
	at com.mirth.connect.donkey.server.channel.Channel.process(Channel.java:1759)
	at com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(Channel.java:1221)
	at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(SourceConnector.java:192)
	at com.mirth.connect.server.controllers.DonkeyEngineController.dispatchRawMessage(DonkeyEngineController.java:1091)
	at com.mirth.connect.connectors.vm.VmDispatcher.send(VmDispatcher.java:157)
	at com.mirth.connect.donkey.server.channel.DestinationConnector.handleSend(DestinationConnector.java:888)
	at com.mirth.connect.donkey.server.channel.DestinationConnector.process(DestinationConnector.java:518)
	at com.mirth.connect.donkey.server.channel.DestinationChain.doCall(DestinationChain.java:121)
	at com.mirth.connect.donkey.server.channel.DestinationChain.call(DestinationChain.java:63)
	at com.mirth.connect.donkey.server.channel.Channel.process(Channel.java:1759)
	at com.mirth.connect.donkey.server.channel.Channel.processSourceQueue(Channel.java:1854)
	at com.mirth.connect.donkey.server.channel.Channel.run(Channel.java:1840)
	at java.base/java.lang.Thread.run(Thread.java:835)Caused by: java.sql.SQLException: Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
	at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2988)
	at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2421)
	at net.sourceforge.jtds.jdbc.TdsCore.isDataInResultSet(TdsCore.java:838)
	at net.sourceforge.jtds.jdbc.JtdsResultSet.<init>(JtdsResultSet.java:149)
	at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:511)
	at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:1029)
	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.getMessageAttachment(JdbcDao.java:1463)
	... 23 more
Reply With Quote
  #2  
Old 10-08-2019, 08:19 AM
cory_cole cory_cole is offline
Mirth Guru
 
Join Date: Mar 2012
Posts: 1,277
cory_cole is on a distinguished road
Default

How are your controlling how message are sent from the receiving channel to the other channels?
Reply With Quote
  #3  
Old 10-08-2019, 08:32 AM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default

I'm not, i just drop them off at the channel. It doesn't matter in what order they end up being send to the PACS.

Have a look... I attached the channels and how they are connected.
This will be used to filter and split series on DICOM level before sending them to the PACS.
If i only use 1 thread, my speed is limited to 5 images/sec. If i ramp this up to 8 or 16 threads it processes an amazing amount of images.
Nothing gets saved in database and pruner is always on to keep the data on disk as low as possible.
Attached Files
File Type: zip MR TEST DICOM Router.zip (66.3 KB, 1 views)
File Type: txt error.txt (13.6 KB, 1 views)

Last edited by V3nn3tj3; 10-08-2019 at 08:36 AM. Reason: Added error
Reply With Quote
  #4  
Old 10-08-2019, 09:13 AM
cory_cole cory_cole is offline
Mirth Guru
 
Join Date: Mar 2012
Posts: 1,277
cory_cole is on a distinguished road
Default

You are. You are filtering based on the value in msg['tag0008103E'].

The next question is, because it is a match and not an =, is there a possibility of the message going to more than one of the senders?
Reply With Quote
  #5  
Old 10-08-2019, 09:21 AM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default

No, if i count the send and filtered it always ends up as the sum of the recieved, so the message is only send to one channel next.
Been trying with less threads but it get the same problem. Doesn't seem like it's a problem with one image, it's random and sometimes happens with one, the more threads, the more error i get.
Reply With Quote
  #6  
Old 10-08-2019, 12:36 PM
cory_cole cory_cole is offline
Mirth Guru
 
Join Date: Mar 2012
Posts: 1,277
cory_cole is on a distinguished road
Default

I think your problem may be that you are removing and storing the attachment. I think that these may be conflicting.
Reply With Quote
  #7  
Old 10-08-2019, 01:12 PM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default

It is wierd, is it not? I will try your suggestion and not remove the attachement after completion of the channel. Will have a look what it does.
But i don't really want to store the attachement either. Is there another way around this? Does it pass a handle to the next channel or the full bytearray of the attachment?
Reply With Quote
  #8  
Old 10-09-2019, 12:03 AM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default

Quote:
Originally Posted by cory_cole View Post
I think your problem may be that you are removing and storing the attachment. I think that these may be conflicting.
When disabling "Remove attachment on completion" the flow works without any problems.
What could be causing this? Is it trying to remove the attachment when still sending the attachment to the next channel? Maybe a wrong race condition? I would expect 1 message to be processed by one thread and only one thread?

Is sending and removing a separate thread in one channel?

I changed the snapshot isolation level on database level, which allows for reads on locked tables. This keeps the mirth server from deadlocking on the database.
Don't know if this is the proper way to fix this.

Last edited by V3nn3tj3; 10-09-2019 at 12:40 AM. Reason: Found some kind of fix...
Reply With Quote
  #9  
Old 10-09-2019, 05:32 AM
cory_cole cory_cole is offline
Mirth Guru
 
Join Date: Mar 2012
Posts: 1,277
cory_cole is on a distinguished road
Default

I believe that the 'remove message' is locking the file and removing. So, when you tried to 'store' the message, it is either locked or not there anymore.
Reply With Quote
  #10  
Old 10-10-2019, 04:49 AM
V3nn3tj3 V3nn3tj3 is offline
Mirth Newb
 
Join Date: Jul 2018
Posts: 8
V3nn3tj3 is on a distinguished road
Default

Quote:
Originally Posted by cory_cole View Post
I believe that the 'remove message' is locking the file and removing. So, when you tried to 'store' the message, it is either locked or not there anymore.
That shouldn't be happening... The problem happens on the destination, so i'm guessing that it is trying to remove the attachment before the destination has finished sending...
Change the isolation level on the database did fix it, and not experiencing any problems.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 08:56 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation