How to Configure a Simple JBoss Cluster in Domain Mode

Clustering is a very important thing to master for any serious user of an application server. Clustering allows for high availability by making your application available on secondary servers when the primary instance is down or it lets you scale up or out by increasing the server density on the host, or by adding servers on other hosts. It can even help to increase performance with effective load balancing between servers based on their respective hardware.

Andy Overton has already covered how to set up a cluster of servers in standalone mode fronted by mod_cluster for load balancing,  so in this post I'll cover clustering in domain mode. I won't rehash mod_cluster settings, so this will just cover the set up of a doman controller on one host, and the host controller and server instances on another host.

To follow along with this blog, you'll need to download either JBoss EAP 6.x or WildFly. I'll be using WildFly 8.2 on Xubuntu 14.04. I'll be using $WF_HOME to refer to your WildFly home directory.

Configuring the Domain Controller 

The domain controller needs both the domain.xml and host.xml configured. In the $WF_HOME/domain/configuration directory, you'll see that those two files are joined by a host-master.xml and a host-slave.xml. These are preconfigured  host.xml files which you can use to give you a head start in making a host.xml for the domain controller (master) and host controller (slave) to use.

You can either change the name of the file to be host.xml, so it will get picked up and used by default, or you can specify the host configuration you want to use on the command line by adding the --host-config argument:

 domain.sh --host-config=host-master.xml   

Whether you choose to modify the host.xml or the host-master.xml, you need to make sure that the empty element <local /> has been added to the <domain-controller> section. 

<domain-controller>  
        <local/>  
     </domain-controller>  

This is so that when WildFly looks to see which server is the domain controller, it knows to become the domain controller itself.

The other change is optional, but recommended. We need to tell the domain controller to bind its management interface to the correct IP address because, by default, it will bind to localhost, so the management communication it needs to do with the remote hosts won't be able to reach the domain controller at all!

We can set this address permanently in the host.xml by making sure the inet-address value is set to the right IP, by changing the 127.0.0.1 in the example below to the correct IP:

 <interfaces>   
      <interface name="management">   
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>   
      </interface>   
    </interfaces>   

The result of that is that the default bind IP of the management interface is no longer localhost, although you can still override this value by starting JBoss with the variable left of the colon as a -D argument:

domain.sh -Djboss.bind.address.management=10.0.0.1  

Next, we need to modify the domain.xml file, where we need to define our server groups; essentially just defining the cluster.

Each server group is named, so we can reference it later, and references a particular profile which needs to be one of the profiles named and defined in the same XML file.

As I mentioned in my previous blog, domain mode has several profiles in the same file (domain.xml) rather than multiple files for each, like standalone mode (standalone.xml, standalone-ha.xml etc.).

In the screenshot, there are two server groups defined - "main-server-group" which references the "full" profile, and "other-server-group" which references the "full-ha" profile. These are just the defaults which come with WildFly, so you're free to use them and modify the settings or create your own from scratch. Whichever you choose, it's a good idea to rename your server group to something meaningful, like a description of the workload, or the application name.

Configuring the Host Controllers

Every host server which you want to be part of the cluster must have the host.xml file configured. We've already configured the host.xml on the domain controller, so now we'll focus on the host controller. Remember, this process can be repeated on any number of hosts, depending on how many servers you want in your server group and their topology.

First, we need to make sure that the domain controller and the host controller can communicate, and to do that we need a valid management user. On the domain controller, run the add-user.sh or  add-user.bat script.

You will need to make sure to: 

• Choose a management user 
• Make sure the user is different than the one you would use to log in to the web console 
• Confirm that the new user will connect one AS process to another AS process 
• Make a note of the secret value (this is very important!) 

You will find that you get prompts similar to the following:

mike@mike-C2B2:~$ /opt/wildfly/wildfly-8.2.0.Final/bin/add-user.sh   
 What type of user do you wish to add?   
  a) Management User (mgmt-users.properties)   
  b) Application User (application-users.properties)   
 (a): a   
 Enter the details of the new user to add.   
 Using realm 'ManagementRealm' as discovered from the existing property files.   
 Username : mgmt   
 Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.   
  - The password should not be one of the following restricted values {root, admin, administrator}   
  - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)   
  - The password should be different from the username   
 Password :   
 Re-enter Password :   
 What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[ ]:   
 About to add user 'mgmt' for realm 'ManagementRealm'   
 Is this correct yes/no? yes   
 Added user 'mgmt' to file '/opt/wildfly/wildfly-8.2.0.Final/standalone/configuration/mgmt-users.properties'   
 Added user 'mgmt' to file '/opt/wildfly/wildfly-8.2.0.Final/domain/configuration/mgmt-users.properties'   
 Added user 'mgmt' with groups to file '/opt/wildfly/wildfly-8.2.0.Final/standalone/configuration/mgmt-groups.properties'   
 Added user 'mgmt' with groups to file '/opt/wildfly/wildfly-8.2.0.Final/domain/configuration/mgmt-groups.properties'   
 Is this new user going to be used for one AS process to connect to another AS process?   
 e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.   
 yes/no? yes   
 To represent the user add the following to the server-identities definition <secret value="bWdtdDEyMyE=" />   

Once we have the secret value for our management user, we can add it to the host.xml file. I'm choosing to modify the host-slave.xml file, since much of the configuration I need is done for me:

<?xml version='1.0' encoding='UTF-8'?>   
  <host xmlns="urn:jboss:domain:2.2">   
    <management>   
      <security-realms>   
        <security-realm name="ManagementRealm">   
          <server-identities>   
             <!-- Replace this with either a base64 password of your own, or use a vault with a vault expression -->   
             <secret value="bWdtdDEyMyE="/>   
          </server-identities>  

Next, we need to tell the host controller where to look for the domain controller. We set this to <local /> for the domain controller's host.xml file, but in the host-slave.xml we have an example <remote> tag filled out for us. All we need to do is add the domain controller's IP or hostname exactly as we did for the management bind address earlier. So our host-slave.xml should go from this:

 <domain-controller>   
      <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>   
   </domain-controller>  

to this:

<domain-controller>  
      <remote host="${jboss.domain.master.address:10.0.0.1}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>  
   </domain-controller>  

This way, like with the management interface on the domain controller, the default address will be 10.0.0.1, but it can also be overridden on the command line if needed.

Once we've sorted the communication out, we need to tell the host controller to actually start some server instances!

At the bottom of the host-slave.xml file, there are two predefined servers to use:

 <servers>  
      <server name="server-one" group="main-server-group"/>  
      <server name="server-two" group="other-server-group">  
        <!-- server-two avoids port conflicts by incrementing the ports in  
           the default socket-group declared in the server-group -->  
        <socket-bindings port-offset="150"/>   
      </server>   
    </servers>  

These are already configured to become members of the two server groups configured in the domain.xml. Note that the second server has to have a port offset. Despite it being in a different server group, it's still going to run on the same host and will attempt to bind to the same ports as the first server unless we tell it not to! We would also need to do the same thing if we added other server instances.

Optionally, we can make things a little easier for ourselves when managing a lot of servers on a lot of hosts. We can give each server instance its own unique name, but we can also name the host by adding a name attribute to the parent tag, changing it from:

<host xmlns="urn:jboss:domain:2.2"> 

to

<host xmlns="urn:jboss:domain:2.2" name="host1">

So both in the logs and in the admin console, you should see this host controller referred to as "host1". Now, if you wanted to name your server instances the same across hosts, you'll be able to tell which is which!

If all you wanted was to configure a single domain controller and a single host controller, then that's all we need to do to get them speaking to each other. You can then carry on and configure mod_cluster and Apache to forward requests on to the right server, or just deploy your applications and connect to them directly.