Using Dynamic Build Agents to Automate Scaling in Jenkins

One instance of Jenkins is fine if you’re running a small project with few developers. But you’ll find, as your team and product grow, that a single instance may not remain stable. When the number of commits increases, so do the processes Jenkins needs to run, and a lone instance’s performance can soon falter and slow your team down.

Thankfully, Jenkins is a scalable platform. Scalability means as your processing needs grow, Jenkins can grow with them.

Scalability with Jenkins sees one instance act as a controller, directing jobs to other instances known as agents. The controller knows the capacity of each agent and will send your builds and tests to the most suitable at the time. By using dynamic build agents, this process can happen automatically, allowing Jenkins to react to your needs.

Thanks to virtual environments like Kubernetes and Amazon Web Services (AWS), you don’t need a finite number of agents or physical hardware. Jenkins, in a dynamic setup, is smart enough to:

• Spin up new agents if none are suitable;
• Prune unused agents;
• Replace corrupted installs.

And all without manual intervention.

In this post, we look at 2 popular ways to set up dynamic scaling from start to finish, with Kubernetes and Amazon Web Services (AWS).

Method 1: Scaling With Kubernetes

Kubernetes is a tool that automatically scales the number of containers needed to keep an application running smoothly. This makes it a good choice for help with scaling your Jenkins instance.

By deploying your Jenkins controller to Kubernetes, your Continuous Integration (CI) setup becomes easier to manage, copy, or recreate if you have problems.

Before You Start

This guide is an example only, and you should experiment with scaling before changing an existing Jenkins setup.

In this example, you set up scalability on a local minikube cluster and use the tools below for configuration. If you're following along, install the tools in the order listed:

1. Docker Desktop – Only needed if you’re on Windows. Make sure Docker Desktop is set to manage Linux containers rather than Windows containers.
2. minikube – Allows you to install Kubernetes clusters on your computer.
3. Chocolatey – Only needed if you’re on Windows. It’s a command-line software management package you use to install Kubectl.
4. Kubectl – A command-line tool to control Kubernetes clusters. Use this Chocolatey command-line to install Kubectl: choco install kubernetes-cli

You can set up scalability with whatever tools you’re used to, but you may need to adjust our instructions slightly.

Step 1: Create a Jenkins Controller Image

First, you must create a dockerfile and use that to build a Jenkins controller image.

Our example dockerfile will create an image that includes Jenkins, plus the Blue Ocean and Kubernetes plugins.

To create the dockerfile and build an image:

1. Create a text file called Dockerfile and add the following Jenkins-suggested script. You can add more plugins to the list if you need – just add their names separated by spaces:

   2. Save and close the file. If using Notepad on Windows, you must remove the ‘.txt’ file extension.

   3. Open a terminal and use the CD command to move to the folder with the file. Use the following command to create the image:

   4. The build will take a little while to process, but after it's complete, you see the image in Docker        Desktop or with the command docker images.

When you create a minikube cluster in the next step, it won't see the image stored locally on your computer as the cluster runs on a virtual environment. To get around this, you can push the image to Docker Hub.

If on Windows, you can do this in Docker Desktop:

1. Click Images on Docker Desktop’s left.
2. Hover your cursor over your image, click the menu icon (3 vertical dots), and select Push to Hub.

Alternatively, you can use the following command line in a terminal:

Step 2: Create a Kubernetes Cluster

Open a terminal window and use the following command to create a Kubernetes cluster:

The cluster may take a while to set up. After it's complete, give it a new namespace to make the cluster easier to use and reference. Use the following command in the terminal to create the namespace 'jenkins':

Step 3: Install Jenkins Using a YAML Deployment File

Copy the following code into a text file and save it as jenkins.yaml. Make sure to change the Image line to point to your image, for example, [Docker username]/[image name].

To deploy the image to the namespace, run the following command from the file’s directory:

The YAML script will now create a Jenkins instance in a Kubernetes pod. The process might take a while, but you can check progress with the following command. It's ready when the Status column reads as Running:

Step 4: Find Your Jenkins Instance URL and Connect to it

Run the following command to find out the instance’s IP address and keep a note of it:

The Ports column will show your instance’s ports in an 8080:54321 format. Take note of the 5 numbers after the colon.

You now combine the minikube IP and the port to make up the URL. For example, if the IP is 123.123.123.123 and the port number is 54321, your Jenkins URL would be http://123.123.123.123:54321.

Go to the URL in your web browser, and you should see the Getting Started screen. This will ask for a one-time administrator password. With a Kubernetes cluster, you need to find this via a command line.

First, you need the name of your Kubernetes pod. Use the following command:

Scroll through the result and find the admin password separated by lines of asterisks. Paste this into your Jenkins instance, and you can complete the setup. See Jenkins documentation for how to set up Jenkins the first time.

Step 5: Get Final Information and Set up the Jenkins Plugin

Now you can set up the plugin in Jenkins. Return to Jenkins in your web browser:

1. Click Manage Jenkins from the menu.
2. Click Manage Nodes and Clouds.
3. Click Configure Clouds.
4. Select Kubernetes from the dropdown, then click Kubernetes Cloud details.
5. Complete the following fields and click Save:
   • Kubernetes URL – enter https://kubernetes.default
   • Kubernetes Namespace – enter jenkins
   • Jenkins URL – enter http://jenkins.jenkins.svc.cluster.local:8080
   • Jenkins Tunnel – enter jenkins-agent.jenkins.svc.cluster.local:50000
6. Scroll to the bottom and click Pod Templates, then Add Pod Template and Pod Template Details.
7. Complete the following fields and click Save:
   • Name – enter jenkins-agent
   • Namespace – enter jenkins
   • Labels – enter jenkins-agent
   • Usage - select Use this node as much as possible from the dropdown

Step 6: Test Everything Is Working

To test Jenkins will scale suitably, you can create some simple build jobs to check how they’re distributed.

First, set Jenkins so it won’t run jobs on the controller (unless you tell it otherwise):

1. Click Manage Jenkins from the menu.
2. Click Configure System.
3. Select Only build jobs with label expressions matching this node from the Usage dropdown, then click Save.

Then create 2 “Hello World” build jobs:

1. Click New Item from the Jenkins dashboard.
2. Enter a suitable name, such as Testing 1, select Freestyle project, and click OK.
3. Under the Build heading, select Execute shell from the dropdown box.
4. Enter echo "Hello World" into the Command box and click Save:
5. Repeat the steps but call your second job Testing 2.

Run both build jobs at the same time. If working correctly, they appear in the Build Queue on Jenkins’ left. During a building job, they appear under the Build Executor Status heading with the 'jenkins-agent' prefix we set earlier. You can also check the build history to double-check exactly where the job has run and if it was successful.

Method 2: Scaling With Amazon Web Services (AWS) and the EC2 Fleet Plugin

Another way to manage Jenkins scalability is with EC2 (Amazon Elastic Compute Cloud) containers and the EC2 Fleet plugin.

This option suits teams who:

• Already have access to AWS;
• Want to easily add scaling to an existing Jenkins instance;
• Prefer to perform setup using a UI rather than command lines.

Though AWS comes at a cost, your financial limits are set at an account level.

Configuring AWS

Sign up to AWS if you don’t have an account, or log into your account if you already have access.

Step 1: Create a Policy

Start by creating a policy to allow access to use AWS EC2 Spot Fleet and the Auto Scaling Group:

1. Click the Services menu at the top, select Security, Identity, & Compliance, then IAM.
2. Click Policies from the left menu under the Access management heading.
3. Click Create policy from the right.
4. Create a new policy with the visual editor to use both EC2 Spot Fleet and Auto Scaling Group. Alternatively, use the JSON editor and paste in the code from the plugin’s setup guide. Click Next: tags.
5. Add tags if needed then click Next: Review.
6. Give the policy a name and description and click Create policy.

Step 2: Create an IAM User With Programmatic Access

Now you create a user with the programmatic access and assign the new policy to it:

1. Click the Services menu at the top, select Security, Identity, & Compliance, and then IAM.
2. Click Users from the left menu under the Access management heading.
3. Click Add users from the right.
4. Give your account a suitable name, select Access key – Programmatic access, and click Next: Permissions.
5. Click the Attach existing policies directly button. Search for the policy you created in the previous step, tick the checkbox to the left and click Next: Tags.
6. Add tags if needed, then click Next: Review.
7. Review your new user and click Create user.

Step 3: Set Credentials to Connect AWS to Jenkins

Next, you create both authentication methods needed to connect AWS to Jenkins.

First, set an access key ID for your newly created IAM user. This will allow Jenkins to see information related to your AWS setup. To set the access key ID:

1. Click the Services menu at the top, select Security, Identity, & Compliance, and then IAM.
2. Click Users from the left menu under the Access management heading.
3. Search for and click the IAM user you created in step 2.
4. Go to the Security credentials tab and click Create access key.
5. Make a note of both the access key ID and the secret access key (click show to see it) and keep it somewhere safe, such as a password manager. You can only see the secret access key at this stage, and you’ll need to create a new one if you lose it.

Now you create a key pair. This allows Jenkins to connect to the instances that AWS will create when scaling. To set a key pair:

1. Click the Services menu at the top, select Compute and click EC2.
2. Click Key Pairs in the left menu under the Network & Security heading.
3. Click Create key pair.
4. Complete the following options and click Create key pair:
   • Name – give the key pair a descriptive name.
   • Key pair type - leave as RSA.
   • Private key file format – select .pem.
   • Tags – add if needed.
5. The private key will automatically download in a text file. Keep this file safe; you need it later.

Step 4: Create an AWS EC2 Spot Fleet or an Auto Scaling Group

You can either create an EC2 Spot Fleet or an Auto Scaling Group. In this example, we created an EC2 Spot Fleet, but check the AWS documentation if you want more info about creating Auto Scaling Groups.

Before starting, you should also have a clear idea about what Amazon Machine Image (AMI) you want to use. An AMI is a pre-built image that includes the operating system and software your EC2 Fleet will create extra machines from.

Your AMI image should include Java 11, as Jenkins won’t scale without it. We use the OpenJDK 11 Java 11 Ubuntu 18.04 AMI from the AWS Marketplace; however,  you could build your own if you need something specific. See AWS’s documentation for more information on AMIs.

To create an EC2 Spot Fleet:

1. Click the Services menu at the top, select Compute and click EC2.
2. Click Spot Requests from the left menu under the Instances heading.
3. Click Request Spot Instances.
4. Set the following options at a minimum – the others you can change as you need:
   • AMI – remember to select an AMI that includes Java 11.
   • Key pair name – select the key pair name we created in step 3.
   • Maintain target capacity – check this box.
5. Scroll to the bottom and click Launch when finished.

It takes a few moments for AWS to create your fleet. Then you can configure Jenkins.

Configuring Jenkins

Step 1: Install the EC2 Fleet Plugin in Jenkins

To install the EC2 Fleet plugin:

1. Click Manage Jenkins from the menu.
2. Click Manage Plugins.
3. Click the Available tab and start typing EC2 Fleet into the Filter field. The plugin should appear in the predicted search results.
4. Check the tick box to the left of the plugin, then click Install without restart.

Jenkins will install the plugin and all dependencies, including other plugins, extensions, and Amazon Software Development Kits (SDKs).

If you haven’t already, you should also install the Credentials Binding plugin.

Step 2: Add the AWS IAM Account and Key Pair to Jenkins

To add your AWS IAM account:

1. Click Manage Jenkins from the menu.
2. Scroll down to the Security heading and click Manage Credentials.
3. Click Jenkins under the Stores scoped to Jenkins heading.
4. Click Global credentials (unrestricted) under the System heading.
5. If no credentials exist, you can click the How about adding some credentials? link, otherwise, click Add Credentials from the left.
6. Complete the following fields and click OK:
   • Kind – Select AWS Credentials from the dropdown.
   • ID – Give the credentials a name you’ll use to identify the credentials in Jenkins.
   • Description – Enter a meaningful description.
   • Access Key ID – Enter the Access Key ID from when you created the IAM account in AWS.
   • Secret Access Key – Enter the access string from when you created the IAM account in AWS.

To add your key pair:

1. Click Manage Jenkins from the menu.
2. Scroll down to the Security heading and click Manage Credentials.
3. Click Jenkins under the Stores scoped to Jenkins heading.
4. Click Global credentials (unrestricted) under the System heading.
5. Click Add Credentials from the left.
6. Complete the following fields and click OK:
   • Kind – Select SSH Username with the private key from the dropdown.
   • ID – Give the credentials a name you’ll use to identify the credentials in Jenkins.
   • Description – Enter a meaningful description.
   • Username – This username is for the AMI you selected.
   • Private Key – Check the Enter directly radio button, click the Add button, and paste in the contents of the key pair file you downloaded earlier.

Step 3: Connect Jenkins to Your AWS EC2 Spot Fleet

Now you connect Jenkins to the AWS EC2 Spot Fleet:

1. Click Manage Jenkins from the menu.
2. Click Manage Nodes and Clouds.
3. Click Configure Clouds.
4. Select Amazon EC2 Fleet from the Add a new cloud dropdown. Complete the following fields and click Save:
   • Name – Enter a descriptive name for use in Jenkins.
   • AWS Credentials – Select your Access Key ID from the AWS Credentials dropdown.
   • Region – Select your fleet’s EC2 fleet’s region from the dropdown.
   • EC2 Fleet – Any EC2 Spot Fleets connected to your Access Key ID will appear here. You may need to click Test Connection if the credentials aren’t showing.
   • Launcher – Select Launch agents via SSH, select the key pair credentials you added in Step 2 and Non-verifying Verification Strategy from both new dropdown boxes.

Step 4: Test Everything Is Working

After a few minutes, go to your Jenkins dashboard. Under the Build Executor Status on the left, you should see both your controller instance and your new EC2 Fleet. If the EC2 fleet is showing any errors, click its name to see logs to identify the problem.

To test that Jenkins will now scale suitably, you can create simple build jobs to check how they’re distributed.

First, you want to set Jenkins so it won’t run jobs on the controller (unless you tell it otherwise):

1. Click Manage Jenkins from the menu.
2. Click Configure System.
3. Select Only build jobs with label expressions matching this node from the Usage dropdown, then click Save.

Now create a “Hello World” build job:

1. Click New Item from the Jenkins dashboard.
2. Enter a suitable name, such as Testing, select Freestyle project, and click OK.
3. Under the Build heading, select Execute shell from the dropdown box.
4. Enter echo "Hello World" into the Command box and click Save:

Run the build job, and if working correctly, you’ll see it appear under your fleet in the Build Executor Status on the left. Once complete, you can also check your build history to check exactly where the job ran.

Changing Your Scaling Options in AWS

After it's set up and working, you can change how you want Jenkins to scale in AWS.

1. Log into AWS.
2. Click the Services menu at the top, select Compute and click EC2.
3. Click Spot Requests from the left menu under the Instances heading.
4. Click the Request ID of your EC2 Spot Fleet.
5. Scroll to the bottom of the page to see the scaling options. Click the Configure button under the Auto Scaling heading.
6. Change settings as you need and save.