Highly Available Docker Registry on AWS With Nexus

Have you ever wondered how you can build a highly available and resilient Docker repository to store your Docker images?

In this post, we will setup an EC2 instance inside a Security Group and create an A record pointing to the server Elastic IP address as follows:

To provision the infrastructure, we will use Terraform as our IaC (Infrastructure as Code) tool. The advantage of using this kind of tool is the ability to spin up a new environment quickly in a different AWS region (or a different IaaS provider) in case of an incident (disaster recovery).

Start by cloning the following GitHub repository:

git clone https://github.com/mlabouardy/terraform-aws-labs.git

Inside the docker-registry folder, update the variables.tfvars with your own AWS credentials (make sure you have the right IAM policies).

resource "aws_instance" "default" {
  ami             = "${lookup(var.amis, var.region)}"
  instance_type   = "${var.instance_type}"
  key_name        = "${aws_key_pair.default.id}"
  security_groups = ["${aws_security_group.default.name}"]

  user_data = "${file("setup.sh")}"

  tags {
    Name = "registry"
  }
}

I specified a shell script to be used as user_data when launching the instance. It will simply install the latest version of Docker CE and turn the instance to Docker Swarm Mode (to benefit from replication and high availability of the Nexus container)

#!/bin/sh
yum update -y
yum install -y docker
service docker start
usermod -aG docker ec2-user
docker swarm init
docker service create --replicas 1 --name registry --publish 5000:5000 --publish 8081:8081 sonatype/nexus3:3.6.2

Note: You can use Configuration Management Tools like Ansible or Chef to provision the server once created.

Then, issue the following command to create the infrastructure:

terraform apply -var-file=variables.tfvars

Once created, you should see the Elastic IP of your instance:

Connect to your instance via SSH:

ssh ec2-user@35.177.167.36

Verify that the Docker Engine is running in Swarm Mode:

Check that the Nexus service is running:

Go back to your AWS Management Console, then, navigate to the Route53 Dashboard. You should see that a new A record has been created that points to the instance IP address.

Point your favorite browser to the Nexus Dashboard URL (registry.slowcoder.com:8081). Log in and create a Docker-hosted registry as below:

Edit the /etc/docker/daemon.json file, it should have the following content:

{
    “insecure-registries” : [“registry.slowcoder.com:5000”]
}

Note: For production, it’s highly recommended that you secure your registry using a TLS certificate issued by a known CA.

Restart Docker for the changes to take effect:

service docker restart

Log into your registry with your Nexus credentials (admin/admin123):

In order to push a new image to the registry:

docker push registry.slowcoder.com:5000/mlabouardy/movies-api:1.0.0-beta

Verify that the image has been pushed to the remote repository:

To pull the Docker image:

docker pull registry.slowcoder.com:5000/mlabouardy/movies-api:1.0.0-beta

Note: Sometimes you end up with many unused and dangling images that can quickly take up a significant amount of disk space:

You can either use the Nexus CLI tool or create a Nexus Task to clean up old Docker Images:

Populate the form as below:

The task above will run every day at midnight to purge unused Docker images from the “mlabouardy” registry.