Mirth Connect
  1. Mirth Connect
  2. MIRTH-1442

If an ack is returned to a channel AFTER the destination timeout has expired, that ack will be matched to the next message queued.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.2
    • Fix Version/s: 2.0.0 Beta 2, 2.0.0
    • Component/s: Administrator
    • Labels:
      None
    • Environment:
      Windows Server 2003, SQL Server 2005, Java 1.6
    • Operating System:
      Windows Server 2003
    • Database:
      Microsoft SQL Server

      Description

      Scenario:
      1. You have an HL7 channel receiving over LLP and then sending over LLP

      2. The destination ack timeout is set to 5000 ms.

      3. A message (Message 1) is sent to the channel

      4. Message 1 is forwarded to the destination

      5. After 5000 ms the receiver has not yet replied

      6. The channel, of course, timesout.

      7. After another 10000 ms the receiver finally acks (Ack 1)

      8. A different message (Message 2) is now sent to the channel (possible, especially if multiple senders are pointed at it)

      9. Message 2 is forwarded to the destination

      10. Without waiting for a reply, the Message 2 is marked as sent, and the channel log indicates that Ack 1 was acknowledged as and Ack for Message 2.

      11. Coincidentally, the destination crashes without processing Message 2

      12. The sender now believes Message 2 was processed, and the destination has no knowledge of its existence!!!!!

      Expected behaviour:
      Ack 1 should be displosed of, and Message 2 should wait for its own ack. Where the message / ack format accomodates, the control ids should be compared for a match. For example, in HL7 2.x, ensure the ack's MSA.2 value matches the sent message's MSH.10 value.

      Of course, longer wait times can reduce the risk.

      Risk of not fixing:
      Patient data can potentially be left unsent, but with the sender believing the opposite to be true.

      1. changes.txt
        0.7 kB
        Denny Lee
      2. server.patch
        1 kB
        Denny Lee

        Activity

        Hide
        Jacob Brauer added a comment -

        Do you have "keep connection open" set to yes?

        Show
        Jacob Brauer added a comment - Do you have "keep connection open" set to yes?
        Hide
        Colin Spalding-Jamieson added a comment -

        Yes, keep connection open is set to yes. Some of our other vendor systems take up to 20 seconds to establish a new connection, so closing it each time is a performance killer.

        I gather you are suggesting that if I set it to no, then the acks will be cleared for each message. However, it would certainly be nice if at the very least, the control id returned in the ack could be matched with the current message. If the match fails, then move onto the next ack. I discovered a case in live where a single message failed to ack in time, and the next 18000 messages were matched with the acks of the preceding message (restarting the service brought things back in line). I can feel fairly certain that the first 17999 messages went through, but it's the last one that I can't be sure of.

        Show
        Colin Spalding-Jamieson added a comment - Yes, keep connection open is set to yes. Some of our other vendor systems take up to 20 seconds to establish a new connection, so closing it each time is a performance killer. I gather you are suggesting that if I set it to no, then the acks will be cleared for each message. However, it would certainly be nice if at the very least, the control id returned in the ack could be matched with the current message. If the match fails, then move onto the next ack. I discovered a case in live where a single message failed to ack in time, and the next 18000 messages were matched with the acks of the preceding message (restarting the service brought things back in line). I can feel fairly certain that the first 17999 messages went through, but it's the last one that I can't be sure of.
        Hide
        Denny Lee added a comment -

        Proposed changes. As suggested, recycle connection if keep connection open is 'yes' and ack was not received within ack timeout.

        Show
        Denny Lee added a comment - Proposed changes. As suggested, recycle connection if keep connection open is 'yes' and ack was not received within ack timeout.
        Hide
        Jacob Brauer added a comment -

        In case the ack times out it is necessary to get a new socket so that the next message does not use the ack of previous message. This was an old fix that needed to be updated to work because the SocketTimeoutException was being caught in LlpProtocol.

        Show
        Jacob Brauer added a comment - In case the ack times out it is necessary to get a new socket so that the next message does not use the ack of previous message. This was an old fix that needed to be updated to work because the SocketTimeoutException was being caught in LlpProtocol.
        Hide
        Mark Maund added a comment -

        Is there any way to fix this issue in Version 1.8.2? I am not ready to use a beta version in a live environment. is the code fragment included in this for v1.8.2? or for v2.0.0?

        Show
        Mark Maund added a comment - Is there any way to fix this issue in Version 1.8.2? I am not ready to use a beta version in a live environment. is the code fragment included in this for v1.8.2? or for v2.0.0?
        Hide
        Jacob Brauer added a comment -

        There isn't currently a patch since there is a workaround of changing keep connection open back to no. If you can't use the workaround I'd suggest contacting the help desk to see if a patch can be created for your specific installation.

        Show
        Jacob Brauer added a comment - There isn't currently a patch since there is a workaround of changing keep connection open back to no. If you can't use the workaround I'd suggest contacting the help desk to see if a patch can be created for your specific installation.
        Hide
        Mark Maund added a comment -

        At one site, I changed "Keep connection open" to NO and I have not had any "Timeout waiting for ACK" errors since. the Control ID is also in sych with the messages. I am glad this seems to be working now but I have a few questions.

        1. What specific fix was implemented in v2.0.0? Comparing Control IDs? or dropping the connection as described?
        2. Why does the "Timeout waiting for ACK" happen so quickly (milli-seconds) when the ACK timeout is set to 5000, 10000, or even 15000ms. Why wouldn't the error log indicate a 5, 10 or 15 second delay before it decides it "timed out".

        Thanks again for your response.

        Show
        Mark Maund added a comment - At one site, I changed "Keep connection open" to NO and I have not had any "Timeout waiting for ACK" errors since. the Control ID is also in sych with the messages. I am glad this seems to be working now but I have a few questions. 1. What specific fix was implemented in v2.0.0? Comparing Control IDs? or dropping the connection as described? 2. Why does the "Timeout waiting for ACK" happen so quickly (milli-seconds) when the ACK timeout is set to 5000, 10000, or even 15000ms. Why wouldn't the error log indicate a 5, 10 or 15 second delay before it decides it "timed out". Thanks again for your response.
        Hide
        Jacob Brauer added a comment -

        The connection is recycled if there is a timeout waiting for ack. This is so that the receiving system cannot send an ACK back to Mirth Connect when Mirth Connect has already marked the message as errored. If it did, then the receiving system would assume there were no issues with Mirth Connect timing out.

        Mirth Connect should respect the ack timeout. If ack timeout happens in a shorter period of time than specified, then the connection was closed by the receiving system or there was a connection error.

        Show
        Jacob Brauer added a comment - The connection is recycled if there is a timeout waiting for ack. This is so that the receiving system cannot send an ACK back to Mirth Connect when Mirth Connect has already marked the message as errored. If it did, then the receiving system would assume there were no issues with Mirth Connect timing out. Mirth Connect should respect the ack timeout. If ack timeout happens in a shorter period of time than specified, then the connection was closed by the receiving system or there was a connection error.

          People

          • Assignee:
            Jacob Brauer
            Reporter:
            Colin Spalding-Jamieson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development