Replicate SEDA Exception for Mule on Your Local Machine

Hello everyone, in this article, I will share how can we replicate a SEDA Exception in our local server in order to resolve this issue.

What Does SEDA Stand For?

Staged Event-Driven Architecture is an approach to software architecture that decomposes complex event-driven architecture into set of stages connected by queue.

What Is SEDA in Mule?

Mule is open source Enterprise Message Bus (ESB) whose model concept is based on the SEDA model. Mule also supports other models, but the SEDA model is the default for Mule. In Mule, each component is treated as a stage with its own thread pool and work queue.

For more reference on the structure, please look at this article.

Why SEDA Exception Occurs

If there is a load of messages processed within the flow in long running process and the queue will become full, which causes a SEDA exception to occur.

For more info, please go to this MuleSoft support article.

Suppose the SEDA Exception occurs in the production environment and you need to fix it. 

For every production issue, the best practice is first to replicate the issue in your local machine. It will help you to resolve the issue more comfortably.

In this article, I will share how I made a mock service with Soup UI, which helps to replicate the exception in your local machine.

You might be thinking, why is there a need to go for mock service; can't I just get the exception in normal request-response format?

The answer to this question is that the SEDA exception occurs internally whenever there is a bulk set of records. This can be possible only when the application runs in production, which processes a huge amount of records per day. Because of the traffic, the queue will get full and this exception takes place. As I already mentioned, by default, Mule follows the SEDA model.

The exception will look like the below:

ERROR 2018-11-15 18:25:59,122 [batch-job-employee-attendance-from-atg-to-hcmBatch-work-manager.13] org.mule.exception.CatchMessagingExceptionStrategy: 
********************************************************************************
Message : The queue for 'SEDA Stage post-atg-attendance-to-hcm.stage1' did not accept new event within 30000 MILLISECONDS.
Payload : {requestNumber=100425110872018091212000020180912200000, requestTimestamp=2018-11-15T18:25:15.184-0500, sourceId=ORA_HWM_WEBCLOCK, timeEvents=[{eventDateTime=2018-09-12T12:00:00.000-05:00, deviceId=100425, supplierDeviceEvent=ORA_HWM_IN, reporterId=11087, reporterIdType=PERSON}, {eventDateTime=2018-09-12T20:00:00.000-05:00, deviceId=100425, supplierDeviceEvent=ORA_HWM_OUT, reporterId=11087, reporterIdType=PERSON}]}
Payload Type : java.util.LinkedHashMap
Element : null @ hcm-atg:null:null
--------------------------------------------------------------------------------
Root Exception stack trace:
org.mule.api.service.FailedToQueueEventException: The queue for 'SEDA Stage post-atg-attendance-to-hcm.stage1' did not accept new event within 30000 MILLISECONDS.
at org.mule.processor.SedaStageInterceptingMessageProcessor.enqueue(SedaStageInterceptingMessageProcessor.java:145)
at org.mule.processor.SedaStageInterceptingMessageProcessor.processNextAsync(SedaStageInterceptingMessageProcessor.java:108)
at org.mule.processor.AsyncInterceptingMessageProcessor.process(AsyncInterceptingMessageProcessor.java:105)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.construct.DynamicPipelineMessageProcessor.process(DynamicPipelineMessageProcessor.java:55)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.interceptor.AbstractEnvelopeInterceptor.processBlocking(AbstractEnvelopeInterceptor.java:59)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:48)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractFilteringMessageProcessor.process(AbstractFilteringMessageProcessor.java:52)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractRequestResponseMessageProcessor.processBlocking(AbstractRequestResponseMessageProcessor.java:57)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:48)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.construct.Flow$1.process(Flow.java:120)
at org.mule.construct.Flow$1.process(Flow.java:115)
at org.mule.execution.ExecuteCallbackInterceptor.execute(ExecuteCallbackInterceptor.java:16)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:35)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:22)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:30)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:67)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.ErrorHandlingExecutionTemplate.execute(ErrorHandlingExecutionTemplate.java:60)
at org.mule.execution.ErrorHandlingExecutionTemplate.execute(ErrorHandlingExecutionTemplate.java:30)
at org.mule.construct.Flow.process(Flow.java:114)
at org.mule.config.spring.factories.FlowRefFactoryBean$2.process(FlowRefFactoryBean.java:264)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.processParts(AbstractMessageSequenceSplitter.java:159)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.process(AbstractMessageSequenceSplitter.java:62)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.routing.Foreach.doProcess(Foreach.java:125)
at org.mule.routing.Foreach.process(Foreach.java:97)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at com.mulesoft.module.batch.engine.BatchProcessingTemplate.process(BatchProcessingTemplate.java:91)
at com.mulesoft.module.batch.engine.buffer.StreamingCommitBuffer$1.process(StreamingCommitBuffer.java:245)
at com.mulesoft.module.batch.engine.buffer.CommitRecordBuffer.doFlush(CommitRecordBuffer.java:85)
at com.mulesoft.module.batch.engine.buffer.StreamingCommitBuffer.flush(StreamingCommitBuffer.java:121)
at com.mulesoft.module.batch.engine.buffer.StreamingCommitBuffer.flushAndForget(StreamingCommitBuffer.java:104)
at com.mulesoft.module.batch.BatchStepCommit.flush(BatchStepCommit.java:118)
at com.mulesoft.module.batch.DefaultBatchStep.flushCommit(DefaultBatchStep.java:373)
at com.mulesoft.module.batch.DefaultBatchStep.finishIfCompleted(DefaultBatchStep.java:336)
at com.mulesoft.module.batch.engine.threading.BatchRecordWork.run(BatchRecordWork.java:101)
at org.mule.work.WorkerContext.run(WorkerContext.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

********************************************************************************

To replicate this, we first have to make a bulk of records around 50k records.

In my application, I made 50k records, which I will mock for my HTTP REST API endpoint.

 <logger message="#[payload]" level="INFO" doc:name="DEBUG Logger"/>
        <until-successful maxRetries="${invoke.maxretries}" millisBetweenRetries="${retry.delay}" synchronous="true" doc:name="Until Successful" >
            <http:request config-ref="Request_Configuration" path="${path}" method="POST" doc:name="HTTP">
                <http:failure-status-code-validator values="500..599"/>
            </http:request>
         </until-successful>
        <object-to-string-transformer mimeType="application/json" doc:name="Object to String"/>
        <logger message="#[payload]" level="DEBUG" category="${tcc.hcm.attendance}" doc:name="DEBUG Logger"/>

To create a mock service in soupUI, you can refer to this article.

Instead of the GET method, we can also do a POST method HTTP call mock service. It all depends on our requirements. In my application, since it was a POST call, I created a POST mock service.

Refer to this mock service for your main Mule project (replace the actual HTTP requester with this mock service) and in soupUI, write a script in the script section for a time delay, which is the main trigger to replicate the SEDA exception.

When we give a time delay, it delays the response from the mock service, which puts the SEDA queue fully on the other side, and there, we go to see our SEDA exception.

The snippet code for the time delay in soupUI is:

<con:afterRequestScript>
 def timeout = 40000 // in milliseconds
 Thread.sleep(timeout)
</con:afterRequestScript>

Once we run our application with this mock service setting, we can easily replicate the SEDA exception to our local machine. Make sure to run bulk records, otherwise, the load will not happen. In my case, I ran 50000 records to achieve this exception.

Once you successfully replicate this, the next thing to do is to resolve it by adding more threads in order to process. Or, the easy way is to make the flow synchronous by processing the strategy where the SEDA exception will occur. Again, it might give a performance issue. Based on your requirement, you choose the method to resolve this.