Using Quartz for Scheduling With MongoDB

I am sure most of us have used the Quartz library to handle scheduled activity within our projects. Although I have interacted with the library quite often in the past, it was the first time I had to use Quartz with MongoDB.

By default, Quartz only provides support for traditional relational databases. Browsing through, I stumbled upon this GitHub repository by Michael Klishin that provides a MongoDB implementation of the Quartz library in a clustered environment.

We will be using a Spring Boot application to show you how we can integrate the Quartz library for scheduling in a clustered environment using MongoDB.

The GitHub repository with the code shown in this article can be found here.

All quartz-related configuration is stored in a property file. The attributes we will use are:

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Quartz Job Scheduling
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~


# Use the MongoDB store
org.quartz.jobStore.class=com.quartz.mongo.intro.quartzintro.scheduler.CustomMongoQuartzSchedulerJobStore


# --- # Note that all the mongo db configuration are set in the CustomMongoQuartzSchedulerJobStore.java class ---
# MongoDB URI (optional if 'org.quartz.jobStore.addresses' is set)
#org.quartz.jobStore.mongoUri=mongodb://localhost:27017

# Comma separated list of mongodb hosts/replica set seeds (optional if 'org.quartz.jobStore.mongoUri' is set)
#org.quartz.jobStore.addresses=localhost


# Will be used to create collections like quartz_jobs, quartz_triggers, quartz_calendars, quartz_locks
org.quartz.jobStore.collectionPrefix=quartz_

# Thread count setting is ignored by the MongoDB store but Quartz requires it
org.quartz.threadPool.threadCount=1

# Skip running a web request to determine if there is an updated version of Quartz available for download
org.quartz.scheduler.skipUpdateCheck=true

org.quartz.jobStore.isClustered=true

#The instance ID will be auto generated by Quartz for all nodes running in a cluster.
org.quartz.scheduler.instanceId=AUTO

org.quartz.scheduler.instanceName=quartzMongoInstance

Let's look at some of these properties. Others are self-explanatory with the comments provided.

• org.quartz.jobStore.class: This defines the job store class that will handle storing job-related details in the database. By default, with the GitHub project mentioned before, we are provided with the MongoDBJobStore. For the purpose of this article, however, we will extend the functionality provided by this class with our own implementation, which will handle the MongoDB configuration based on Spring profiles. 
• org.quartz.jobStore.mongoUri: You will define the comma-separated MongoDB URIs here if you want to use the default MongoDBJobStore class. With this implementation, however, since we are defining a custom job store, we will not be using this property. An example of how you would define this would be mongodb://<ip1>:<port>,<ip2>:<port>.
• org.quartz.jobStore.collectionPrefix: This property defines the prefix for the collections created for the purpose of storing Quartz-specific details.

Let's first see how our JobStore configuration class looks:

package com.quartz.mongo.intro.quartzintro.scheduler;

import org.apache.commons.lang3.StringUtils;
import org.quartz.impl.StdSchedulerFactory;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.io.ClassPathResource;

import com.novemberain.quartz.mongodb.MongoDBJobStore;
import com.quartz.mongo.intro.quartzintro.constants.SchedulerConstants;
import com.quartz.mongo.intro.quartzintro.constants.SystemProperties;

/**
 * 
 * <p>
 * We extend the {@link MongoDBJobStore} because we need to set the custom mongo
 * db parameters. Some of the configuration comes from system properties set via
 * docker and the others come via the application.yml files we have for each
 * environment.
 * </p>
 * 
 * < These are set as part of initialization. This class is initialized by
 * {@link StdSchedulerFactory} and defined in the quartz.properties file.
 * 
 * </p>
 * 
 * @author dinuka
 *
 */
public class CustomMongoQuartzSchedulerJobStore extends MongoDBJobStore {

 private static String mongoAddresses;
 private static String userName;
 private static String password;
 private static String dbName;
 private static boolean isSSLEnabled;
 private static boolean isSSLInvalidHostnameAllowed;

 public CustomMongoQuartzSchedulerJobStore() {
  super();
  initializeMongo();
  setMongoUri("mongodb://" + mongoAddresses);
  setUsername(userName);
  setPassword(password);
  setDbName(dbName);
  setMongoOptionEnableSSL(isSSLEnabled);
  setMongoOptionSslInvalidHostNameAllowed(isSSLInvalidHostnameAllowed);
 }

 /**
  * <p>
  * This method will initialize the mongo instance required by the Quartz
  * scheduler.
  * 
  * The use case here is that we have two profiles;
  * </p>
  * 
  * <ul>
  * <li>Development</li>
  * <li>Production</li>
  * </ul>
  * 
  * <p>
  * So when constructing the mongo instance to be used for the Quartz
  * scheduler, we need to read the various properties set within the system
  * to determine which would be appropriate depending on which spring profile
  * is active.
  * </p>
  * 
  */
 private static void initializeMongo() {
  /**
   * The use case here is that when we run our application, the property
   * spring.profiles.active is set as a system property during production.
   * But it will not be set in a development environment.
   */
  String env = System.getProperty(SystemProperties.ENVIRONMENT);
  env = StringUtils.isNotBlank(env) ? env : "dev";
  YamlPropertiesFactoryBean commonProperties = new YamlPropertiesFactoryBean();
  commonProperties.setResources(new ClassPathResource("application.yml"));
  /**
   * The mongo DB user name and password are only password as command line
   * parameters in the production environment and for the development
   * environment it will be null which is why we use
   * StringUtils#trimToEmpty so we can pass empty strings for the user
   * name and password in the development environment since we do not have
   * authentication on the development environment.s
   */
  userName = StringUtils.trimToEmpty(commonProperties.getObject().getProperty(SystemProperties.SERVER_NAME));
  password = StringUtils.trimToEmpty(System.getProperty(SystemProperties.MONGO_PASSWORD));
  dbName = commonProperties.getObject().getProperty(SchedulerConstants.QUARTZ_SCHEDULER_DB_NAME);

  YamlPropertiesFactoryBean environmentSpecificProperties = new YamlPropertiesFactoryBean();

  userName = commonProperties.getObject().getProperty(SystemProperties.SERVER_NAME);

  switch (env) {
  case "prod":
   environmentSpecificProperties.setResources(new ClassPathResource("application-prod.yml"));
   /**
    * By deafult, in the production mongo instance, SSL is enabled and
    * SSL invalid host name allowed property is set.
    */
   isSSLEnabled = true;
   isSSLInvalidHostnameAllowed = true;
   mongoAddresses = environmentSpecificProperties.getObject().getProperty(SystemProperties.MONGO_URI);
   break;
  case "dev":
   /**
    * For the development profile, we just read the mongo URI that is
    * set.
    */
   environmentSpecificProperties.setResources(new ClassPathResource("application-dev.yml"));
   mongoAddresses = environmentSpecificProperties.getObject().getProperty(SystemProperties.MONGO_URI);
   break;

  }

 }

}

In the above implementation, we have retrieved the MongoDB details pertaining to the active profile. If no profile is defined, it defaults to the development profile. We have used the YamlPropertiesFactoryBean here to read off the application properties pertaining to different environments.

Moving on, we then need to let Spring manage the creation of the Quartz configuration using the SchedulerFactoryBean. 

package com.quartz.mongo.intro.quartzintro.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.scheduling.quartz.SchedulerFactoryBean;

/**
 * This class will configure and setup quartz using the
 * {@link SchedulerFactoryBean}
 * 
 * @author dinuka
 *
 */
@Configuration
public class QuartzConfiguration {

 /**
  * Here we integrate quartz with Spring and let Spring manage initializing
  * quartz as a spring bean.
  * 
  * @return an instance of {@link SchedulerFactoryBean} which will be managed
  *         by spring.
  */
 @Bean
 public SchedulerFactoryBean schedulerFactoryBean() {
  SchedulerFactoryBean scheduler = new SchedulerFactoryBean();
  scheduler.setApplicationContextSchedulerContextKey("applicationContext");
  scheduler.setConfigLocation(new ClassPathResource("quartz.properties"));
  scheduler.setWaitForJobsToCompleteOnShutdown(true);
  return scheduler;
 }

}

We define this as a Configuration class so that it will be picked up when we run the Spring Boot application.

The call to the setApplicationContextSchedulerContextKey method here is in order to get a reference to the Spring application context within our job class, which is as follows:

package com.quartz.mongo.intro.quartzintro.scheduler.jobs;

import org.quartz.DisallowConcurrentExecution;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.PersistJobDataAfterExecution;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.core.env.Environment;
import org.springframework.scheduling.quartz.QuartzJobBean;
import org.springframework.scheduling.quartz.SchedulerFactoryBean;

import com.quartz.mongo.intro.quartzintro.config.JobConfiguration;
import com.quartz.mongo.intro.quartzintro.config.QuartzConfiguration;

/**
 * 
 * This is the job class that will be triggered based on the job configuration
 * defined in {@link JobConfiguration}
 * 
 * @author dinuka
 *
 */
@PersistJobDataAfterExecution
@DisallowConcurrentExecution
public class SampleJob extends QuartzJobBean {

 private static Logger log = LoggerFactory.getLogger(SampleJob.class);

 private ApplicationContext applicationContext;

 /**
  * This method is called by Spring since we set the
  * {@link SchedulerFactoryBean#setApplicationContextSchedulerContextKey(String)}
  * in {@link QuartzConfiguration}
  * 
  * @param applicationContext
  */
 public void setApplicationContext(ApplicationContext applicationContext) {
  this.applicationContext = applicationContext;
 }

 /**
  * This is the method that will be executed each time the trigger is fired.
  */
 @Override
 protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
  log.info("This is the sample job, executed by {}", applicationContext.getBean(Environment.class));

 }
}

As you can see, we get a reference to the application context when the SchedulerFactoryBean is initialized. The part of the Spring documentation I would like to draw your attention to is as follows:

In case of a QuartzJobBean, the reference will be applied to the Job instance as bean property. An "applicationContext" attribute will correspond to a "setApplicationContext" method in that scenario.

Next, we configure the job to be run with the frequency by which to run the scheduled activity.

package com.quartz.mongo.intro.quartzintro.config;

import static org.quartz.TriggerBuilder.newTrigger;

import java.time.LocalDateTime;
import java.time.ZoneId;
import java.util.Date;

import javax.annotation.PostConstruct;

import org.quartz.JobDetail;
import org.quartz.JobKey;
import org.quartz.SimpleScheduleBuilder;
import org.quartz.Trigger;
import org.quartz.TriggerKey;
import org.quartz.impl.JobDetailImpl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.quartz.SchedulerFactoryBean;

import com.quartz.mongo.intro.quartzintro.constants.SchedulerConstants;
import com.quartz.mongo.intro.quartzintro.scheduler.jobs.SampleJob;

/**
 * 
 * This will configure the job to run within quartz.
 * 
 * @author dinuka
 *
 */
@Configuration
public class JobConfiguration {

 @Autowired
 private SchedulerFactoryBean schedulerFactoryBean;

 @PostConstruct
 private void initialize() throws Exception {
  schedulerFactoryBean.getScheduler().addJob(sampleJobDetail(), true, true);
  if (!schedulerFactoryBean.getScheduler().checkExists(new TriggerKey(
    SchedulerConstants.SAMPLE_JOB_POLLING_TRIGGER_KEY, SchedulerConstants.SAMPLE_JOB_POLLING_GROUP))) {
   schedulerFactoryBean.getScheduler().scheduleJob(sampleJobTrigger());
  }

 }

 /**
  * <p>
  * The job is configured here where we provide the job class to be run on
  * each invocation. We give the job a name and a value so that we can
  * provide the trigger to it on our method {@link #sampleJobTrigger()}
  * </p>
  * 
  * @return an instance of {@link JobDetail}
  */
 private static JobDetail sampleJobDetail() {
  JobDetailImpl jobDetail = new JobDetailImpl();
  jobDetail.setKey(
    new JobKey(SchedulerConstants.SAMPLE_JOB_POLLING_JOB_KEY, SchedulerConstants.SAMPLE_JOB_POLLING_GROUP));
  jobDetail.setJobClass(SampleJob.class);
  jobDetail.setDurability(true);
  return jobDetail;
 }

 /**
  * <p>
  * This method will define the frequency with which we will be running the
  * scheduled job which in this instance is every minute three seconds after
  * the start up.
  * </p>
  * 
  * @return an instance of {@link Trigger}
  */
 private static Trigger sampleJobTrigger() {
  return newTrigger().forJob(sampleJobDetail())
    .withIdentity(SchedulerConstants.SAMPLE_JOB_POLLING_TRIGGER_KEY,
      SchedulerConstants.SAMPLE_JOB_POLLING_GROUP)
    .withPriority(50).withSchedule(SimpleScheduleBuilder.repeatMinutelyForever())
    .startAt(Date.from(LocalDateTime.now().plusSeconds(3).atZone(ZoneId.systemDefault()).toInstant()))
    .build();
 }

}

There are many ways you can configure your scheduler, including cron configurations. For the purpose of this article, we will define a simple trigger to run every minute, three seconds after startup. We define this as a Configuration class so that it will be picked up when we run the Spring Boot application.

That's about it. When you now run the Spring Boot application class found in the GitHub repository with a running MongoDB instance, you will see the following collections created:

• quartz_calendars 
  
• quartz_jobs 
  
• quartz_locks 
  
• quartz_schedulers 
  
• quartz_triggers
  

Thank you for reading. If you have any comments, improvements, or suggestions, do kindly leave a comment.