Scheduling Jobs Using PCF Scheduler

Anyone who has tried to run a batch application either through Spring Cloud Task or Spring Batch Job on PCF knows the pain of running the application on it and doing scheduling. To date, PCF has been considered only as a means to run "web" process types of applications, where there is no room for "Batch" Process types, as there was no built-in scheduler available. But, with PCF 2.0, a very nice service called "PCF Scheduler" was introduced. This can help with the running of a Batch Application, as per the defined schedule.

• Scheduler for PCF is a service for scheduling the execution of Diego tasks, such as database migrations, emails, or batch jobs, as well as the execution of outbound HTTP calls.
• It is a Marketplace Service which has to be bound to a PCF application and then, using it, scheduled jobs and calls can be defined to run for this application.
• Commands for the jobs will be run on the VM where the PCF application is running, which basically limits the functionality of it to be around the application only.

What You Will Build Here

• Create, run, and schedule jobs and view job history.
• Create, run, and schedule calls and view call history.

What You Will Need

• JDK 1.8
• Gradle
• CF CLI
• Scheduler for PCF CLI plugin on your local workstation (required for the create call, as it's available only through the CF CLI). This plugin is required to execute the commands related to the create/run/schedule call. It can be downloaded from Pivotal Network (https://network.pivotal.io/products/p-scheduler/). The command to install plugin is as follows:

cf install-plugin scheduler-for-pcf-cliplugin-windows32-exe-1.1.0

1. How to Create Jobs Using Scheduler

• Get a sample Spring Boot application. You can use this Spring Cloud Task Batch application (https://github.com/RajeshBhojwani/spring-batch-job.git).
• Push this application to PCF. The above sample app has a manifest.yml file, so cf push will be sufficient to push the app to PCF.
• Once the application starts running, stop the application as it will crash after running through all the steps of the Batch Job.
• Now a job can be created on this application using PCF Scheduler.

• Click on "ENABLE SCHEDULING" under the Tasks tab of the application. This will bind the PCF Scheduler with the application.
• This will enable the button, "CREATE JOB." This button can be used to create a job in the application.

• Fill in all 3 fields with the required details.
  • Job Name - This can be any meaningful name.
  • Command - Accepts only Linux commands.
  • Cron Expressions - Accepts the following format only: "MIN HOUR DAY-OF-MONTH        MONTH DAY-OF-WEEK"
• You can test this by clicking on the"Run Now" option also.
• The same steps can be followed to work with the CF CLI as well. Run the below commands

cf create-job batch-example ReLaunch #".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher"
cf schedule-job ReLaunch #"20 9 ? * * "
cf run-job ReLaunch  #(To Test the job)

• You can see the Status of the job and its history on the same page.

This completes the creation of the job on any existing application.

2. How to Create a Call Using Scheduler

• The create-call command is available only through the CF CLI.
• Get a sample Spring Boot application. You can use this REST web service application (https://github.com/RajeshBhojwani/spring-rest-service.git).
• Push this application to PCF. The above sample app has a manifest.yml file, so cf push will be sufficient to push the app to PCF.
• Now, a call can be created for this application.
• Run the below commands on the CF CLI to do a create-call

cf bind-service gs-rest-service scheduler-dev #(scheduler-dev is the instance for scheduler-for-pcf service)

cf create-call gs-rest-service callgreeting #https://rest-service.apps.np.sdppcf.com/greetings  (This will create a call on the application to make outbound call to one of the endpoint)

cf run-call callgreeting  #(This is to test the call if its working fine or not. Check the Logs to see if traffic flow is ok or any error)

cf schedule-call callgreeting #"0 12 ? * * "  (This will schedule it as per cron setting)

This completes the creation of a call on any existing application.

Use Cases

There is a batch application which needs to be scheduled on a specific time every day or every month.                       

1. Create a batch app.
2. Bind this app with PCF Scheduler and create a job to relaunch the app.

Follow the instructions mentioned in "How to Create Jobs using Scheduler." A REST endpoint has to be scheduled to run at a particular time. This might be a  one-off asynchronous call. The ideal flow will look like the below picture.

• 

• Create a Batch App.

• Bind this app with PCF Scheduler and Create a call using the CF CLI and schedule it.

Follow the instructions for the create-call mentioned in "How to Create a Call Using Schedule" section.

Limitations

1. PCF Scheduler is a market service so it always needs an existing app in PCF with which it has to be bound.
2. Create Job can run only Linux commands, so serves only a limited function.
3. It doesn't support MASSL calls using the create-call option.
4. The Cron Job command does not support the option to schedule at secondary levels.
5. If a job fails, there is no re-run option
6. There is no alerting or notification feature yet for the status of the job.

Known Issues

1. There is no check if duplicate jobs are created.
2. If one job is already running as per your schedule and the same job is executed manually at the same time, there is no check for it.

Food for Thought

1. Is PaaS the right choice for a batch job kind of application?
2. Should PaaS be allowed to be accessed from outside to get a batch job scheduled?
3. Are there any security vulnerabilities?