Service Discovery for NGINX Plus with ZooKeeper

Our previous blog posts about service discovery with Consul and etcd discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. In such systems it’s common to assign network locations to service instances dynamically, because their locations can change over time due to autoscaling, failures, or upgrades. To track the current network location of a service instance, clients need a sophisticated service discovery mechanism.

In this blog, we’ll explain how to dynamically add or remove load-balanced servers that are registered with Apache ZooKeeper^TM, a tool used for service discovery, in combination with NGINX Plus ’on-the-fly reconfiguration API. With this solution you can automate service discovery and change the set of load-balanced servers without having to reload the NGINX Plus configuration.

Apache ZooKeeper is a software project of the Apache Software Foundation that provides an open source distributed configuration service, synchronization service, and naming registry for large distributed systems. ZooKeeper nodes – or Znodes, as they are popularly called – store their data in a hierarchical name space, much like a file system or a tree data structure. All clients can read from and write to the nodes, making ZooKeeper a shared configuration service.

To make it easier to combine the on-the-fly reconfiguration API with ZooKeeper, we’ve created a samplezookeeper-demo, with step-by-step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker,Docker Compose, and Homebrew, you can spin up a Docker-based environment in which NGINX Plus load balances HTTP traffic to a couple of hello-world applications, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

• ZooKeeper – Performs service discovery.
• Registrator – Registers services with ZooKeeper. Registrator monitors the starting and stopping of containers and updates ZooKeeper about the state changes.
• hello – Simulates a backend server. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
• A second instance of the hello app – Simulates another backend server.
• NGINX Plus – Load balances the above services.

The NGINX Plus container listens on the public port 80, and the built-in NGINX Plus dashboard on port 8080. The ZooKeeper container listens on ports 2181, 2888, and 3888.

Registrator monitors Docker for new containers that are launched with exposed ports, and registers the associated services with ZooKeeper. By setting environment variables within the containers, we can be more explicit about how to register the services with ZooKeeper. For each hello-world container, we set the SERVICE_TAGS environment variable to production to identify the container as an upstream server for NGINX Plus to load balance. When a container quits or is removed, Registrator removes its corresponding Znode entry from ZooKeeper automatically.

Finally, with a tool written in Ruby and included in the sample demo, zk-tool, we use ZooKeeper watchesto trigger an external handler (script.sh) every time there is a change in the list of registered service containers. This bash script gets the list of all current NGINX Plus upstream servers, uses zk-tool to loop through all the containers registered with ZooKeeper that are tagged production, and uses the on-the-fly reconfiguration (upstream_conf) API to add them to the NGINX Plus upstream group if they’re not listed already. It also then removes from the NGINX Plus upstream group any production-tagged containers that are not registered with ZooKeeper.

Summary

Using a script like the one in our demo to dynamically reconfigure upstream groups based on the services registered with ZooKeeper automates upstream reconfiguration in NGINX Plus. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in ZooKeeper and its addition or removal within the upstream group in NGINX Plus.

Check out our other blogs about for server discovery for NGINX Plus:

• Service Discovery for NGINX Plus with Consul
• Service Discovery for NGINX Plus with etcd