How To Create Asynchronous and Retryable Methods With Failover Support
Learn about a new framework that allows processing methods asynchronously with retries in case of failure and the support of load-balancing and failover.
Join the DZone community and get the full member experience.
Join For FreeWhile developing an application, we need to make some processing more robust and less fault-tolerant, especially when requesting remote services that may remain down for a long duration.
In this article, we will introduce a new framework that aims to provide a declarative non-blocking retry support for methods in Spring-based applications using annotations.
This framework has two main implementations:
- Thread pool task-based implementation: This implementation is based on
ThreadPoolTaskExecutor
without keeping the task executor thread busy during the whole retry processing, unlike the combination of@Async
(see "Spring Boot - Async methods") and@Retryable
(see "Retry Handling With Spring-Retry"). Indeed, when using the Spring traditionalretry
annotation, the thread that runs the method performs the whole retry policy, including waiting periods, and remains busy until the end. For example, if theThreadPoolTaskExecutor
has 10 threads with a retry policy that may take 5 minutes, and the application receives 30 requests, only 10 requests can be processed simultaneously and the others will be blocked for the whole 5 minutes. So the execution of the 30 requests may take 15 minutes. - Quartz job-based implementation: This implementation is based on the Quartz library. It supports load-balancing, failover, and persistence if configured with JDBC JobStore. This means that even if a node in the cluster is down, the others can take over the operation and perform the retries.
Basic Concepts
Annotation
In order to make a method retryable, you can annotate it with @AsyncRetryable
and specify the following attributes:
retryPolicy
(mandatory): The policy bean name that defines the next retry time if an exception has been thrown during the method executionretryFor
(optional): List of the exceptions for which the retry should be performednoRetryFor
(optional): List of the exceptions for which the retry should not be performedretryListener
(optional): The listener bean name that triggers events related to the annotated method execution
The following example shows how to use @AsyncRetryable
in a declarative style:
@Bean
public class Service {
@AsyncRetryable(retryPolicy = "fixedWindowRetryableSchedulingPolicy",
retryFor = IOException.class,
noRetryFor={ArithmeticException.class},
retryListener = "retryListener")
public void method(String arg1, Object arg2){
// ...do something
}
}
In this example, if an exception of type IOException
is thrown during the method perform()
execution, the retry will be made according to the policy fixedWindowRetryableSchedulingPolicy
. If an exception of type ArithmeticException
is thrown, no retry will be made.
All the events that happened during the method call are reported to the bean listener retryListener
.
Retry Policy
The retry policy defines the next execution time for each failing execution. Indeed, when the annotated method throws a retryable exception, the retry policy bean is called in order to get the period to wait before calling the method again.
This framework provides three basic implementations:
FixedWindowRetryableSchedulingPolicy
: This policy is used for retries with a fixed waiting period and a max attempt limit.StaticAsyncRetryableSchedulingPolicy
: This policy accepts an array of waiting periods for each attempt.LinearAsyncRetryableSchedulingPolicy
: This policy multiplies each time the previous waiting period by a coefficient in order to increase the duration of the next one. The coefficient default value is 2.
The following example shows how to configure a FixedWindowRetryableSchedulingPolicy
that will trigger the annotated method for the first time in 10 seconds, then make 3 retries within a waiting period of 20 seconds each.
@Configuration
public class AsyncRetryConfiguration {
...
@Bean
public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
}
}
Remark: It is possible to customize the retry policy by implementing the interface AsyncRetryableSchedulingPolicy
.
Retry Listener
The retry listener is used to detect events during the retry processing life cycle. The AsyncRetryableListener
interface is defined as below:
public interface AsyncRetryableListener<T> {
void beforeRetry(Integer retryCount, Object[] args);
void afterRetry(Integer retryCount,T result, Object[] args, Throwable e);
void onRetryEnd(Object[] args, Throwable e);
}
The methods beforeRetry()
and afterRetry()
are triggered respectively before and after the call of the annotated method. The method onRetryEnd()
is triggered at the end of the retry process. The methods defined above are called nether the annotated method succeeds or fails.
The methods' attributes are:
retryCount
: The current retry numberresult
: The value returned by the annotated method in case of successargs
: The annotated method argument values in the same ordere
: The thrown exception during the execution of the annotated method; this value is null if the method is executed with success.
How To Use It
Thread Pool Task-Based Implementation
In order to use the asynchronous retry feature based on the Spring Thread pool task scheduler, all you have to do is to add the following dependency:
<dependency>
<artifactId>async-retry-spring-scheduler</artifactId>
<groupId>org.digibooster.retryable</groupId>
<version>1.0.2</version>
</dependency>
Add the annotation EnableThreadPoolBasedAsyncRetry
to a configuration class, and finally, define the retry policy bean as follow:
@Configuration
@EnableThreadPoolBasedAsyncRetry
public class AsyncRetryConfiguration {
/**
* the annotated method will be triggered the first time after 1 second and will
* perform 2 retries eatch 20 seconds in case of failure
*/
@Bean
public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
}
}
Quartz-Based Implementation (In Memory)
This configuration uses the RAM in order to store the retry jobs. It is not persistent and doesn't support load-balancing and failover. So retries will be lost if the server restarts.
In order to use this implementation, add the following dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
</dependency>
<dependency>
<artifactId>async-retry-quartz-scheduler</artifactId>
<groupId>org.digibooster.retryable</groupId>
<version>1.0.2</version>
</dependency>
Add a configuration class that extends DefaultQuartzBasedAsyncRetryableConfigAdapter
:
@Configuration
public class RetryAsyncQuartzInMemoryConfiguration extends DefaultQuartzBasedAsyncRetryableConfigAdapter {
@Bean
public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
}
@Bean("schedulerFactoryBean")
public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
@Autowired QuartzProperties quartzProperties) {
SchedulerFactoryBean factory = new SchedulerFactoryBean();
Properties properties = new Properties();
properties.putAll(quartzProperties.getProperties());
factory.setQuartzProperties(properties);
factory.setJobFactory(quartzSchedulerJobFactory);
return factory;
}
}
Finally, add the following lines to the application.yml file:
spring:
quartz:
auto-startup: true
job-store-type: memory
Quartz-Based Implementation (JDBC)
This configuration uses the database in order to store the retry jobs. It is persistent and supports load-balancing and failover, so retries will not be lost if the server restarts.
In order to use this implementation, add the following dependencies:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
</dependency>
<dependency>
<artifactId>async-retry-quartz-scheduler</artifactId>
<groupId>org.digibooster.retryable</groupId>
<version>1.0.2</version>
</dependency>
Add a configuration class that extends QuartzDBBasedAsyncRetryableConfigAdapter
as follows:
@Configuration
public class ConfigurationClass extends QuartzDBBasedAsyncRetryableConfigAdapter {
@Autowired
PlatformTransactionManager transactionManager;
@Override
public PlatformTransactionManager getTransactionManager() {
return transactionManager;
}
@Bean
public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
}
@Bean("schedulerFactoryBean")
public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
@Autowired QuartzProperties quartzProperties,
@Autowired DataSource dataSource) {
SchedulerFactoryBean factory = new SchedulerFactoryBean();
Properties properties = new Properties();
properties.putAll(quartzProperties.getProperties());
factory.setQuartzProperties(properties);
factory.setJobFactory(quartzSchedulerJobFactory);
factory.setDataSource(dataSource);
return factory;
}
}
Finally, add the following configuration to the application.yml file:
spring:
quartz:
auto-startup: true
job-store-type: jdbc
properties:
org.quartz.jobStore.isClustered: true
org.quartz.scheduler.instanceName: RetryInstance # optional
org.quartz.scheduler.instanceId: AUTO # optional
jdbc:
initialize-schema: always # optional
Remark: When using Quartz with a database, the retry will be executed with a delay due to Quartz implementation. To decrease the delay, you can change the value of the property org.quartz.jobStore.clusterCheckinInterval
. The framework source code is published on GitHub.
Opinions expressed by DZone contributors are their own.
Comments