Network error IOException: Address alread in use

[csuscheck]

I'm getting an intermittent error:
Network error IOException: Address already in use: connect (com.webreach.mirth.server.MirthJavascriptTransfor merException).

My Mirth version is 1.7.1.3422

We're reading files (file reader) from a directory and pushing them to a channel writer. This is a stress test so there are 1000's of files.
the dashboard lists 14371 received, 13851 sent, 519 errored.

Any ideas on what to look for ?


ERROR-412: VM Connector error
ERROR MESSAGE: Error routing message
org.mule.umo.routing.RoutingException: Failed to transform message before applying the filter. Failed to route event via endpoint: ImmutableMuleEndpoint{connector=com.webreach.mirth .connectors.vm.VMConnector@b8edf5, endpointUri=vm://fdb2f7a0-bf96-4745-8126-4df9892d2f2b, transformer=Transformer{name='fdb2f7a0-bf96-4745-8126-4df9892d2f2b_source_preprocessor', returnClass=false, returnClass=false, sourceTypes=[]}, name='fdb2f7a0-bf96-4745-8126-4df9892d2f2b', type='receiver', properties={}, transactionConfig=org.mule.impl.MuleTransactionCon fig@1d30ac7, filter=null, deleteUnacceptedMessages=false, initialised=true, securityFilter=null, synchronous=null, initialState=started, createConnector=0}. Message payload is of type: java.lang.String
at org.mule.routing.inbound.SelectiveConsumer.isMatch (SelectiveConsumer.java:64)
at org.mule.routing.inbound.InboundMessageRouter.rout e(InboundMessageRouter.java:79)
at org.mule.providers.AbstractMessageReceiver$Default InternalMessageListener.onMessage(AbstractMessageR eceiver.java:493)
at org.mule.providers.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:272)
at org.mule.providers.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:231)
at com.webreach.mirth.connectors.vm.VMMessageReceiver .dispatchMessage(VMMessageReceiver.java:118)
at com.webreach.mirth.connectors.vm.VMMessageDispatch er.routeTemplatedMessage(VMMessageDispatcher.java: 216)
at com.webreach.mirth.connectors.vm.VMMessageDispatch er.doSend(VMMessageDispatcher.java:273)
at org.mule.providers.AbstractMessageDispatcher.send( AbstractMessageDispatcher.java:166)
at org.mule.impl.MuleSession.sendEvent(MuleSession.ja va:191)
at org.mule.impl.MuleSession.sendEvent(MuleSession.ja va:130)
at org.mule.routing.outbound.AbstractOutboundRouter.s end(AbstractOutboundRouter.java:85)
at org.mule.routing.outbound.FilteringMulticastingRou ter.route(FilteringMulticastingRouter.java:89)
at org.mule.routing.outbound.OutboundMessageRouter$1. doInTransaction(OutboundMessageRouter.java:78)
at org.mule.transaction.TransactionTemplate.execute(T ransactionTemplate.java:48)
at org.mule.routing.outbound.OutboundMessageRouter.ro ute(OutboundMessageRouter.java:82)
at org.mule.impl.model.DefaultMuleProxy.onCall(Defaul tMuleProxy.java:247)
at org.mule.impl.model.seda.SedaComponent.doSend(Seda Component.java:209)
at org.mule.impl.model.AbstractComponent.sendEvent(Ab stractComponent.java:277)
at org.mule.impl.MuleSession.sendEvent(MuleSession.ja va:201)
at org.mule.routing.inbound.InboundMessageRouter.send (InboundMessageRouter.java:161)
at org.mule.routing.inbound.InboundMessageRouter.rout e(InboundMessageRouter.java:128)
at org.mule.providers.AbstractMessageReceiver$Default InternalMessageListener.onMessage(AbstractMessageR eceiver.java:493)
at org.mule.providers.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:272)
at org.mule.providers.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:231)
at com.webreach.mirth.connectors.file.FileMessageRece iver.processFile(FileMessageReceiver.java:236)
at com.webreach.mirth.connectors.file.FileMessageRece iver.poll(FileMessageReceiver.java:138)
at org.mule.providers.PollingMessageReceiver.run(Poll ingMessageReceiver.java:83)
at org.mule.impl.work.WorkerContext.run(WorkerContext .java:290)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.runTask(ThreadPoolExecutor. java:650)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :675)
at java.lang.Thread.run(Unknown Source)
Caused by: org.mule.umo.transformer.TransformerException:
CHANNEL: PM_Generic_MFN_toMyWay_Main
CONNECTOR: sourceConnector
SCRIPT SOURCE:
LINE NUMBER: 33
DETAILS: Wrapped java.sql.SQLException: Network error IOException: Connection refused: connect (com.webreach.mirth.server.MirthJavascriptTransfor merException)
at com.webreach.mirth.server.mule.transformers.JavaSc riptTransformer.evaluateScript(JavaScriptTransform er.java:448)
at com.webreach.mirth.server.mule.transformers.JavaSc riptTransformer.transform(JavaScriptTransformer.ja va:346)
at org.mule.transformers.AbstractEventAwareTransforme r.doTransform(AbstractEventAwareTransformer.java:4 8)
at org.mule.transformers.AbstractTransformer.transfor m(AbstractTransformer.java:197)
at org.mule.transformers.AbstractTransformer.transfor m(AbstractTransformer.java:200)
at org.mule.impl.MuleEvent.getTransformedMessage(Mule Event.java:251)
at org.mule.routing.inbound.SelectiveConsumer.isMatch (SelectiveConsumer.java:61)
... 31 more
Caused by: com.webreach.mirth.server.MirthJavascriptTransform erException:
CHANNEL: PM_Generic_MFN_toMyWay_Main
CONNECTOR: sourceConnector
SCRIPT SOURCE:
LINE NUMBER: 33
DETAILS: Wrapped java.sql.SQLException: Network error IOException: Connection refused: connect
at com.webreach.mirth.server.mule.transformers.JavaSc riptTransformer.evaluateScript(JavaScriptTransform er.java:439)
... 37 more
Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(C onnectionJDBC2.java:372)
at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(C onnectionJDBC3.java:50)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.ja va:178)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at com.webreach.mirth.server.util.DatabaseConnection. <init>(DatabaseConnection.java:64)
at com.webreach.mirth.server.util.DatabaseConnectionF actory.createDatabaseConnection(DatabaseConnection Factory.java:65)
at sun.reflect.GeneratedMethodAccessor357.invoke(Unkn own Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Un known Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.mozilla.javascript.MemberBox.invoke(MemberBox. java:142)
at org.mozilla.javascript.NativeJavaMethod.call(Nativ eJavaMethod.java:201)
at org.mozilla.javascript.optimizer.OptRuntime.callN( OptRuntime.java:84)
at org.mozilla.javascript.gen.c194._c12(82f7d7d2-6917-493d-ae52-06d3a254fd56:38)
at org.mozilla.javascript.gen.c194.call(82f7d7d2-6917-493d-ae52-06d3a254fd56)
at org.mozilla.javascript.optimizer.OptRuntime.callNa me(OptRuntime.java:95)
at org.mozilla.javascript.gen.c194._c15(82f7d7d2-6917-493d-ae52-06d3a254fd56:79)
at org.mozilla.javascript.gen.c194.call(82f7d7d2-6917-493d-ae52-06d3a254fd56)
at org.mozilla.javascript.optimizer.OptRuntime.callNa me0(OptRuntime.java:106)
at org.mozilla.javascript.gen.c194._c0(82f7d7d2-6917-493d-ae52-06d3a254fd56:133)
at org.mozilla.javascript.gen.c194.call(82f7d7d2-6917-493d-ae52-06d3a254fd56)
at org.mozilla.javascript.ContextFactory.doTopCall(Co ntextFactory.java:337)
at org.mozilla.javascript.ScriptRuntime.doTopCall(Scr iptRuntime.java:2755)
at org.mozilla.javascript.gen.c194.call(82f7d7d2-6917-493d-ae52-06d3a254fd56)
at org.mozilla.javascript.gen.c194.exec(82f7d7d2-6917-493d-ae52-06d3a254fd56)
at com.webreach.mirth.server.mule.transformers.JavaSc riptTransformer.evaluateScript(JavaScriptTransform er.java:393)
... 37 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.reflect.GeneratedMethodAccessor355.invoke(Unkn own Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Un known Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocke tForJDBC3(SharedSocket.java:289)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(Shar edSocket.java:250)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(C onnectionJDBC2.java:297)
... 62 more

[dans]

looks like you are not closing some db connection properly in your transformer

[csuscheck]

The system doesn't have any transformers, so I don't close a db connection (don't create one!)

The error only happens with some of the files, not all.... seems odd. Seems like it could be a networking issue.

[dans]

so there are no transformers, filters, or preprocessor/postprocessor scripts? Are you doing any type of database connection or tcp connection anywhere?

[csuscheck]

There are no there are no transformers, filters, or preprocessor/postprocessor scripts. There is no type of database connection or tcp connection anywhere.

We are using a channel writer, so I can try "Wait for Channel Response". Since there's a lot of files being read and written to the channel, perhaps the receiving channel is causing errors by trying to multitask.

Any other ideas?

[dans]

Does the receiving channel have any database or tcp connections? Any transformers or scripts of any sort? The stack trace indicates there is some kind of connection problem in channel "PM_Generic_MFN_toMyWay_Main".

[csuscheck]

We have a feeder channel that reads an ftp site (file) and then passes it to the main channel via the channel writer. The idea is to have multiple ways from various customers to get the data into a main channel - which has transformations and database activity. The scheme works great with TCPIP instead of a file reader in the feeder channel. I would guess that the problem happens when the feeder reads a file and passes the data to the main channel which cannot respond until it's done with the file, but the feeder tries to pass another file into the main channel. We'll try the "wait for channel" option on the channel writer from the feeder to see if this takes care of the problem - the code itself may not be a problem (of course our code is perfect).