If a user places a non-serializable object in the connector or channel map and persistent queuing is off, then the message will process just fine. The database will still be able to store it too (in most cases) because toString() is called rather than attempting to serialize it.
However, if persistent queuing is on, then the message will always fail with an exception like this:
ERROR-408: MLLP Connector error
ERROR MESSAGE: Can't save payload to queue
java.io.NotSerializableException: <class name here>