JobRunr + Kubernetes + Terraform
Deploy the JobRunr application to a Kubernetes cluster on the Google Cloud Platform (GCP) using Terraform.
·
Join the DZone community and get the full member experience.
Join For FreeIn this new tutorial, we will build further upon on our first tutorial — Easily process long-running jobs with JobRunr — and deploy the JobRunr application to a Kubernetes cluster on the Google Cloud Platform (GCP) using Terraform. We then scale it up to 10 instances to have a whopping 869% speed increase compared to only one instance!
This tutorial is a beginners guide on the topic cloud infrastructure management. Feel free to skip to the parts that interest you.
Kubernetes, also known as k8s, is the hot new DevOps tool for deploying high-available applications. Today, there are a lot of providers all supporting Kubernetes including the well known Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), and Amazon Elastic Kubernetes Service (EKS).
Although the world is currently in difficult times because of COVID-19, Acme Corp (see the first tutorial) hired so many people that there are now about 10.000 employees working for them. Acme Corp's CEO insists that all employees get their weekly salary slip before Sunday 11pm but this has now become impossible - the amount of time it takes for generating that many salary slips is just too long.
Luckily, JobRunr is here to help as it is a distributed background job processing framework. In this tutorial we will:
- Create a Docker image from our
SalarySlipMicroserviceJobRunr application using Jib by Google - Upload the Docker image to a private Docker registry at Google
- Use Terraform to define our infrastructure as code which includes a Google Cloud Sql instance.
- Deploy a Kubernetes cluster using Terraform to Google Cloud
- Deploy one instance of the
SalarySlipMicroserviceJobRunr Docker image to the Kubernetes cluster - Start generating all the employee slips
- Scale to 10 instances of the
SalarySlipMicroserviceJobRunr application and all of this without any change to our production java code!
TLDR; you can find the complete project on our GitHub repository: https://github.com/jobrunr/example-salary-slip/tree/kubernetes
Postgres as Database
In the first version of our application, we used an embedded H2 Database. As we now go for a deployment on Google Cloud Platform (GCP), we will use a Cloud Sql Postgres instance. To do so, we need to change our DataSource in the SalarySlipMicroService as follows:
public DataSource dataSource() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(String.format("jdbc:postgresql:///%s", System.getenv("DB_NAME")));
config.setUsername(System.getenv("DB_USER"));
config.setPassword(System.getenv("DB_PASS"));
config.addDataSourceProperty("socketFactory", "com.google.cloud.sql.postgres.SocketFactory");
config.addDataSourceProperty("cloudSqlInstance", System.getenv("CLOUD_SQL_INSTANCE"));
return new HikariDataSource(config);
}
Dockerize It!
Since Kubernetes runs Pods - which are in fact one or more Docker Containers - we first need to create a Docker Image from our application. Jib is a tool from Google to easily create Docker images from your Java application using only Maven or Gradle.
In our build.gradle file, we add the following plugin:
xxxxxxxxxx
plugins {
...
id 'com.google.cloud.tools.jib' version '2.2.0'
}
...
jib {
from {
image = "gcr.io/distroless/java:11"
}
to {
image = "gcr.io/jobrunr-tutorial-kubernetes/jobrunr-${project.name}:1.0"
}
container {
jvmFlags = ["-Duser.timezone=Europe/Brussels"]
ports = ["8000", "8080"]
}
}
jib plugin and tell it to build further upon the distroless Java 11 base image. We tag it with gcr.io/jobrunr-tutorial-kubernetes/jobrunr-${project.name}:1.0 so that it will be available later in GCP, specify the timezone and tell it to expose some ports.
If we now run the gradle command: ./gradlew jibDockerBuild it will create a new Docker image for us, ready to run on Docker!
Install the Necessary Tools
We now need to install all the necessary tools and create a Google Cloud account:
- Google Cloud SDK: Google Cloud SDK is a set of tools that you can use to manage resources and applications hosted on Google Cloud Platform.
- Kubectl: Kubectl is a command line tool for controlling Kubernetes clusters.
- Terraform: Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision a data center infrastructure using a high-level configuration language known as Hashicorp Configuration Language.
The installation for these tools is well explained and differs for each OS. Follow the installation guide for them and come back to the tutorial once you have done so.
We also need an account for Google Cloud. Using your browser navigate to https://console.cloud.google.com/ — when you first login to the Google Cloud Platform you get 300 € of free credit, more than enough for us. You can activate it on the top right.
The Google console dashboard with the free trial at the top
Create the GCP Project
In this tutorial, we will use the terminal as much as possible - so fire up a terminal and login to gcloud using the command: ~$ gcloud auth login - this will allow you to login only once for all future gcloud commands.
To deploy a Kubernetes cluster to GCP, we first need to create a new GCP project, add a billing account to it, enable the container API's and upload our docker image:
xxxxxxxxxx
~$ gcloud projects create jobrunr-tutorial-kubernetes --name="JobRunr K8s Tutorial" --set-as-default
~$ gcloud beta billing accounts list
~$ gcloud beta billing projects link jobrunr-tutorial-kubernetes --billing-account ${accountId}
~$ gcloud services enable container.googleapis.com
~$ gcloud services enable sqladmin.googleapis.com
~$ docker push gcr.io/jobrunr-tutorial-kubernetes/jobrunr-example-paycheck:1.0
We also need a Terraform service account with the necessary rights to create the Kubernetes cluster in the GCP project.
xxxxxxxxxx
~$ gcloud iam service-accounts create terraform --display-name "Terraform admin account"
~$ gcloud projects add-iam-policy-binding jobrunr-tutorial-kubernetes --member='serviceAccount:terraform@jobrunr-tutorial-kubernetes.iam.gserviceaccount.com' --role='roles/editor'
~$ gcloud projects add-iam-policy-binding jobrunr-tutorial-kubernetes --member='serviceAccount:terraform@jobrunr-tutorial-kubernetes.iam.gserviceaccount.com' --role='roles/resourcemanager.projectIamAdmin'
~$ gcloud projects add-iam-policy-binding jobrunr-tutorial-kubernetes --member='serviceAccount:terraform@jobrunr-tutorial-kubernetes.iam.gserviceaccount.com' --role='roles/cloudsql.client'
~$ gcloud iam service-accounts keys create ~/.config/gcloud/jobrunr-tutorial-kubernetes-terraform-admin.json --iam-account=terraform@jobrunr-tutorial-kubernetes.iam.gserviceaccount.com
~$ export TF_CREDS=~/.config/gcloud/jobrunr-tutorial-kubernetes-terraform-admin.json
~$ export GOOGLE_APPLICATION_CREDENTIALS=${TF_CREDS}
editor, resourcemanager.projectIamAdmin and cloudsql.client. Finally, a private key is created which is saved to a json file and exported so that it can be used by Terraform.
Terraform Deep Dive
Now we're all setup, we can start defining our infrastructure as code using Terraform.
In Terraform, several concepts exist:
- Providers: a provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services
- Resources: resources are the most important element in the Terraform language. Each resource block describes one or more infrastructure objects, such as virtual networks, compute instances
- Variables: a variable can have a default value. if you omit the default value, Terraform will ask you to provide it when running a terraform command
- Modules: a module is nothing more than a folder which combines related terraform files
- Outputs: sometimes a variable is needed which is only known after terraform has done a change on a cloud provider — think of ip-addresses that are given to your application. An output takes that value and exposes it to your variables
In Terraform, you can organize your code the anyway you like it - Terraform itself figures out how to deploy it. In this tutorial, we will use two modules:
- gke module: this module is responsible for setting up a Kubernetes Cluster and a Postgres CloudSql instance.
- k8s module: this module will deploy our application to the Kubernetes Cluster and expose it to the internet via a service.
Our entry point in Terraform is the main.tf configuration file. Next to it, are two directories: gke and k8s. The final directory layout is as follows:
- gke
- variables.tf
- gcp.tf
- cluster.tf
- cloudsql.tf
- k8s
- variables.tf
- k8s.tf
- deployments.tf
- services.tf
- main.tf
Entrypoint for Terraform - main.tf
main.tf is the entry point in our infrastructure as code.
xxxxxxxxxx
#####################################################################
# Variables
#####################################################################
variable "project" {
default = "jobrunr-tutorial-kubernetes"
}
variable "region" {
default = "europe-west1"
}
variable "username" {
default = "admin"
}
variable "password" {
default = "cluster-password-change-me"
}
#####################################################################
# Modules
#####################################################################
module "gke" {
source = "./gke"
project = var.project
region = var.region
username = var.username
password = var.password
}
module "k8s" {
source = "./k8s"
host = module.gke.host
username = var.username
password = var.password
client_certificate = module.gke.client_certificate
client_key = module.gke.client_key
cluster_ca_certificate = module.gke.cluster_ca_certificate
cloudsql_instance = module.gke.cloudsql_db_instance
cloudsql_db_name = module.gke.cloudsql_db_name
cloudsql_db_user = module.gke.cloudsql_db_user
cloudsql_db_password = module.gke.cloudsql_db_password
}
k8s module reuses outputs from the gke module.
GKE Module
Our GKE module will create a container cluster on Google Cloud and provision a Postgres Cloud Sql instance. We start by defining some variables that can then be used in the other Terraform files.
xxxxxxxxxx
#####################################################################
# GKE Variables
#####################################################################
variable "project" {}
variable "region" {}
variable "username" {}
variable "password" {}
main.tf file.
xxxxxxxxxx
#####################################################################
# GKE Provider
#####################################################################
provider "google" {
project = var.project
region = var.region
}
xxxxxxxxxx
#####################################################################
# GKE Cluster
#####################################################################
resource "google_container_cluster" "jobrunr-tutorial-kubernetes" {
name = "jobrunr-tutorial-kubernetes"
location = var.region
initial_node_count = 1
master_auth {
username = var.username
password = var.password
}
node_config {
machine_type = "n1-standard-2"
oauth_scopes = [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/trace.append",
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/cloud-platform", //needed for sqlservice
"https://www.googleapis.com/auth/sqlservice.admin"
]
}
}
#####################################################################
# Output for K8S
#####################################################################
output "client_certificate" {
value = google_container_cluster.jobrunr-tutorial-kubernetes.master_auth[0].client_certificate
sensitive = true
}
output "client_key" {
value = google_container_cluster.jobrunr-tutorial-kubernetes.master_auth[0].client_key
sensitive = true
}
output "cluster_ca_certificate" {
value = google_container_cluster.jobrunr-tutorial-kubernetes.master_auth[0].cluster_ca_certificate
sensitive = true
}
output "host" {
value = google_container_cluster.jobrunr-tutorial-kubernetes.endpoint
sensitive = true
}
xxxxxxxxxx
#####################################################################
# GKE Cloud SQL
#####################################################################
resource "google_sql_database_instance" "postgres" {
database_version = "POSTGRES_11"
settings {
tier = "db-g1-small"
database_flags {
name = "max_connections"
value = 100
}
}
timeouts {
delete = "10m"
}
}
resource "google_sql_user" "users" {
name = "jobrunr"
instance = google_sql_database_instance.postgres.name
password = "changeme"
}
resource "google_sql_database" "database" {
name = "jobrunr"
instance = google_sql_database_instance.postgres.name
}
#####################################################################
# Output for K8S
#####################################################################
output "cloudsql_db_name" {
value = google_sql_database.database.name
sensitive = true
}
output "cloudsql_db_user" {
value = google_sql_user.users.name
sensitive = true
}
output "cloudsql_db_password" {
value = google_sql_user.users.password
sensitive = true
}
output "cloudsql_db_instance" {
value = "${var.project}:${var.region}:${google_sql_database_instance.postgres.name}"
sensitive = true
}
k8s Module
The k8s module will deploy our docker image we created earlier on and provide it with the environment variables to connect to the Postgres Cloud Sql instance. It will also create a Kubernetes service to expose the application via an Ingress load-balancer to the internet.
We again start with the variables that can be used in the other Terraform files from the k8s module.
xxxxxxxxxx
#####################################################################
# K8S Variables
#####################################################################
variable "username" {}
variable "password" {}
variable "host" {}
variable client_certificate {}
variable client_key {}
variable cluster_ca_certificate {}
variable cloudsql_instance {}
variable cloudsql_db_name {}
variable cloudsql_db_user {}
variable cloudsql_db_password {}
main.tf file which acts as a bridge between the gke module and k8s module.
xxxxxxxxxx
#####################################################################
# K8S Provider
#####################################################################
provider "kubernetes" {
host = var.host
username = var.username
password = var.password
client_certificate = base64decode(var.client_certificate)
client_key = base64decode(var.client_key)
cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
}
xxxxxxxxxx
#####################################################################
# K8S Deployment
#####################################################################
resource "kubernetes_deployment" "jobrunr-tutorial" {
metadata {
name = "jobrunr"
labels = {
app = "jobrunr"
}
}
spec {
replicas = 1
selector {
match_labels = {
app = "jobrunr"
}
}
template {
metadata {
labels = {
app = "jobrunr"
}
}
spec {
container {
image = "gcr.io/jobrunr-tutorial-kubernetes/jobrunr-example-paycheck:1.0"
name = "jobrunr"
port {
container_port = 8000
}
port {
container_port = 8080
}
env {
name = "CLOUD_SQL_INSTANCE"
value = var.cloudsql_instance
}
env {
name = "DB_NAME"
value = var.cloudsql_db_name
}
env {
name = "DB_USER"
value = var.cloudsql_db_user
}
env {
name = "DB_PASS"
value = var.cloudsql_db_password
}
resources {
limits {
cpu = "0.5"
memory = "1024Mi"
}
requests {
cpu = "250m"
memory = "512Mi"
}
}
}
}
}
}
}
xxxxxxxxxx
#####################################################################
# K8S Service
#####################################################################
resource "kubernetes_service" "jobrunr-tutorial" {
metadata {
name = "jobrunr-tutorial"
}
spec {
selector = {
app = kubernetes_deployment.jobrunr-tutorial.spec.0.template.0.metadata[0].labels.app
}
port {
name = "dashboard"
port = 8000
target_port = 8000
}
port {
name = "rest-api"
port = 8080
target_port = 8080
}
type = "LoadBalancer"
}
}
Deploy Time!
We now can use Terraform commands to provision our application to the Google Cloud Platform. Make sure you are in the directory which contains the main.tf file and the gke and k8s folders when issuing the following commands:
xxxxxxxxxx
~/jobrunr/gcloud$ terraform init
~/jobrunr/gcloud$ terraform plan
~/jobrunr/gcloud$ terraform apply
terraform init command downloads the necessary plugins (google and kubernetes) to execute the requested infrastructure changes. The second command, terraform plan lists all the required infrastructure changes. The last command, terraform apply makes the actual infrastructure changes.
After you run the terraform apply command you have to wait... typical deploy time is about 5 minutes.
After the deployment succeeds, we can query kubernetes to find out the public ip-address.
xxxxxxxxxx
~/jobrunr/gcloud$ gcloud container clusters get-credentials jobrunr-tutorial-kubernetes --region europe-west1
~/jobrunr/gcloud$ kubectl get services
~/jobrunr/gcloud$ kubectl get pods
kubectl get pods lists the pods - there should be one pod active.
Testing Time...
Since the salary slip microservice is now available on the internet, we can test it. First, we will create 10.000 employees in our database. To do so, fire up your favorite browser and go to the url http://${public-ip-from-the-service}:8080/create-employees?amount=10000. This takes about 15 seconds.
Now, visit the JobRunr dashboard - you can find it at http://${public-ip-from-the-service}:8000/dashboard. Navigate to the Recurring jobs tab and trigger the 'Generate and send salary slip to all employees' job. After about 15 seconds, you should have 10.000 enqueued jobs. Let's measure how long it takes to process them...
Scale it up!
Now, let's add 10 instances of our application to the cluster by changing the replica attribute in the deployment.tf file.
xxxxxxxxxx
#####################################################################
# K8S Deployment
#####################################################################
resource "kubernetes_deployment" "jobrunr-tutorial" {
metadata {
...
}
spec {
replicas = 10
...
}
}
We now apply this change again using the Terraform apply command:~/jobrunr/gcloud$ terraform apply
If you run the command ~/jobrunr/gcloud$ kubectl get pods you will now see 10 pods running our JobRunr application. Let's trigger the 'Generate and send salary slip to all employees' recurring job again and wait for it to finish.
It only took 1.292 seconds or 21 minutes and 30 seconds!
To keep your free credit for GCP, do not forget to issue the command terraform destroy. It will stop all pods, remove the Kubernetes cluster and delete the Postgres Cloud Sql instance.
Conclusion
JobRunr can easily scale horizontally and allows to distribute all long-running background jobs over multiple instances without any change to the Java code. In an ideal world, we would have seen a 900% speed increase instead of the 869% we see now as we added 9 extra pods. As JobRunr only performs each job only once, there is some overhead when pulling jobs from the queue explaining the difference.
Learn more
I hope you enjoyed this tutorial and you can see the benefits of JobRunr, Terraform, and Kubernetes — it allows you to easily scale horizontally and distribute all long-running background jobs over multiple instances without any change to the Java code.
To learn more, check out these guides:
- JobRunr — Java batch processing made easy...
- Terraform — Provision servers in the cloud with Terraform
- Kubernetes — Getting started with Kubernetes
- Jib — Create fast and easy docker images with Jib
Published at DZone with permission of Ronald Dehuysser. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments