ZooKeeper for Microservice Registration and Discovery

In a microservice world, multiple services are typically distributed in a PaaS environment. Immutable infrastructure is provided by containers or immutable VM images. Services may scale up and down based upon certain pre-defined metrics. The exact address of the service may not be known until the service is deployed and ready to be used.

The dynamic nature of the service endpoint address is handled by service registration and discovery. In this, each service registers with a broker and provides more details about itself, such as the endpoint address. Other consumer services then query the broker to find out the location of a service and invoke it. There are several ways to register and query services such as ZooKeeper, etcd, consul, Kubernetes, Netflix Eureka and others.

Monolithic to Microservice Refactoring showed how to refactor an existing monolith to a microservice-based application. User, Catalog, and Order service URIs were defined statically. This blog will show how to register and discover microservices using ZooKeeper.

Many thanks to Ioannis Canellos (@iocanel) for all the ZooKeeper hacking!

What is ZooKeeper?

ZooKeeper is an Apache project and provides a distributed, eventually consistent hierarchical configuration store.

 

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

So a service can register with ZooKeeper using a logical name, and the configuration information can contain the URI endpoint. It can consists of other details as well, such as QoS.

 ZooKeeper has a steep learning curve as explained in Apache ZooKeeper Made Simpler with Curator. So, instead of using ZooKeeper directly, this blog will use Apache Curator.

Curator n ˈkyoor͝ˌātər: a keeper or custodian of a museum or other collection – A ZooKeeper Keeper.

Apache Curator has several components, and this blog will use the Framework:

The Curator Framework is a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations.

ZooKeeper Concepts

ZooKeeper Overview provides a great overview of the main concepts. Here are some of the relevant ones:

• Znodes: ZooKeeper stores data in a shared hierarchical namespace that is organized like a standard filesystem. The name space consists of data registers – called znodes, in ZooKeeper parlance – and these are similar to files and directories.
• Node name: Every node in ZooKeeper’s name space is identified by a path. Exact name of a node is a sequence of path elements separated by a slash (/).
• Client/Server: Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the client will connect to a different server.
• Configuration data: Each node in a ZooKeeper namespace can have data associated with it as well as children. ZooKeeper was originally designed to store coordination data, so the data stored at each node is usually small, in less than KB range).
• Ensemble: ZooKeeper itself is intended to be replicated over a sets of hosts called an ensemble. The servers that make up the ZooKeeper service must all know about each other.
• Watches: ZooKeeper supports the concept of watches. Clients can set a watch on a znode. A watch will be triggered and removed when the znode changes.

ZooKeeper is a CP system with regards to CAP theorem. This means if there is a partition failure, it will be consistent but not available. This can lead to problems that are explained in Eureka! Why You Shouldn’t Use ZooKeeper for Service Discovery.

Nevertheless, ZooKeeper is one of the most popular service discovery mechanisms used in the microservices world.

Lets get started!

Start ZooKeeper

Start a ZooKeeper instance in a Docker container:

docker run -d -p 2181:2181 fabric8/zookeeper

Verify ZooKeeper instance by using telnet as:

telnet $(docker-machine ip dockerhost)

Type the command “ruok” to verify that the server is running in a non-error state. The server will respond with “imok” if it is running:

Trying 192.168.99.103...
Connected to dockerhost.
Escape character is '^]'.
ruok
imokConnection closed by foreign host.

Otherwise it will not respond at all. ZooKeeper has other similar four-letter commands.

Service Registration and Discovery

Each service, User, Catalog, and Order in our case, has an eagerly initialized bean that registers and unregisters the service as part of lifecycle initialization methods. Here is the code from CatalogService:

@Inject @ZooKeeperRegistry ServiceRegistry services;

private static final String endpointURI = "http://localhost:8080/catalog/resources/catalog";
private static final String serviceName = "catalog";

@PostConstruct
public void registerService() {
    services.registerService(serviceName, endpointURI);
}

@PreDestroy
public void unregisterService() {
    services.unregisterService(serviceName, endpointURI);
}

The code is pretty simple, it injects ServiceRegistry class, with @ZooKeeperRegistry qualifier. This is then used to register and unregister the service. Multiple URIs, one each for a stateless service, can be registered under the same logical name.

At this time, the qualifier comes from another maven module. A cleaner Java EE way would be to move the @ZooKeeperRegistry qualifier to a CDI extension (#20). And when this qualifier when specified on any REST endpoint will register the service with ZooKeeper (#22). For now, service endpoint URI is hardcoded as well (#24).

What does ZooKeeper class look like?

1. ZooKeeper class uses constructor injection and hardcoding IP address and port (#23):

2. @ApplicationScoped
   public class ZooKeeper implements ServiceRegistry {
   
       private final CuratorFramework curatorFramework;
       private final ConcurrentHashMap<String, String> uriToZnodePath;
   
       @Inject
       public ZooKeeper() {
           try {
               Properties props = new Properties();
               props.load(this.getClass().getResourceAsStream("/zookeeper.properties"));
   
               curatorFramework = CuratorFrameworkFactory
                       .newClient(props.getProperty("host") 
                               + ":" 
                               + props.getProperty("port"), new RetryNTimes(5, 1000));
               curatorFramework.start();
               uriToZnodePath = new ConcurrentHashMap<>();
           } catch (IOException ex) {
               throw new RuntimeException(ex.getLocalizedMessage());
           }
       }

   It does the following tasks:

   1. Loads ZooKeeper’s host/port from a properties file
   2. Initializes Curator framework and starts it
   3. Initializes a hashmap to store the URI name to zNode mapping. This node is deleted later to unregister the service.
3. Service registration is done using registerService method as:

4. String znode = "/services/" + name;
   
   if (curatorFramework.checkExists().forPath(znode) == null) {
       curatorFramework.create().creatingParentsIfNeeded().forPath(znode);
   }
   
   String znodePath = curatorFramework
           .create()
           .withMode(CreateMode.EPHEMERAL_SEQUENTIAL)
           .forPath(znode+"/_", uri.getBytes());
   
   uriToZnodePath.put(uri, znodePath);

   Code is pretty straight forward:

   1. Create a parent zNode, if needed
   2. Create an ephemeral and sequential node
   3. Add metadata, including URI, to this node
5. Service discovery is done using discover method as:

String znode = "/services/" + name;

List<String> uris = curatorFramework.getChildren().forPath(znode);
return new String(curatorFramework.getData().forPath(ZKPaths.makePath(znode, uris.get(0))));

Again, simple code:

1. Find all children for the path registered for the service
2. Get metadata associated with this node, URI in our case, and return.The first such node is returned in this case. Different QoS parameters can be attached to the configuration data. This will allow to return the appropriate service endpoint.

Read ZooKeeper Javadocs for API.

ZooKeeper watches can be setup to inform the client about the lifecycle of the service (#27). ZooKeeper path caches can provide an optimized implementation of the children nodes (#28).

Multiple Service Discovery Implementations

Our shopping cart application has two two service discovery implementations – ServiceDisccoveryStatic and ServiceDiscoveryZooKeeper. The first one has all the service URIs defined statically, and the other one retrieves them from ZooKeeper.

Other means to register and discover can be easily added by creating a new package in services module and implementing ServiceRegistry interface. For example, Snoop, etcd, Consul, and Kubernetes. Feel free to send a PR for any of those.

Run Application

1. Make sure the ZooKeeper image is running as explained earlier.
2. Download and run WildFly:

3. ./bin/standalone.sh

4. Deploy the application:

5. cd microservice
   mvn install

6. Access the application at localhost:8080/everest-web/. Learn more about the application and different components in Monolithic to Microservices Refactoring for Java EE Applications blog.