Run Axon Server on Kubernetes or Virtual Machines?

Running applications built with Axon Framework on Kubernetes is a no-brainer, but is Kubernetes a good choice for running Axon Server with the wide choice of deployment options and environments available, andwhat factors do you need to consider when deploying Axon Server?

Whether you’re developing a greenfield event-driven microservices application or you’re taking your existing monolithic application and "strangling" it into one, Axon Framework paired with Axon Server works well together to handle both scenarios.

Since Axon Framework’s existence stretches much further into the past than Axon Server, the majority of users already have an existing Axon Framework deployment. A lot of these users have sought to leverage Kubernetes to obtain a unified approach to managing their microservices and have reported success running Axon Framework applications on Kubernetes.

Having successfully tried Axon Server following the release of Axon 4.0 and having an existing deployment of Axon Framework on Kubernetes, it is a tempting and natural progression to deploy Axon Server there as well. However, this might not be the wisest choice because of the fundamental differences in workloads between the two.

In this blog, we discuss these workload differences and consider the two leading options available as well as the considerations needed for each one.

Application Microservices vs. Database Components

The clearest way to see the differences in workload characteristics is with a table:

There are many considerable differences here but they result from a fundamental difference in needs. We want to leverage the benefits of microservices to increase the modularity and scalability our applications. Since these are the benefits we are after, these components are usually small and are stateless to allow for easy scaling and fast development iterations.

On the other hand, the underlying database usually doesn't require that level of change, which is particularly true for event-sourced applications where changes to the event’s schema are handled by the application itself. Whilst scaling is possible with Axon Server it is unwise to scale it as rapidly as you might an application microservice.

So how does this apply to the two leading options?

Kubernetes

# Its tempting to manage all your services like this:

$ kubectl apply -f kubernetes/axonserver.yaml
statefulset.apps "axonserver" created
service "axonserver-gui" created
service "axonserver" created

As you may already know, Kubernetes “automates the deployment, scaling and management of containerized applications” and Kubernetes has proven itself with respect to stateless workloads. Kubernetes does a lot of the heavy lifting in running applications and, as mentioned already, if you're using Kubernetes already to run your applications, it is tempting to run Axon Server there as well.

However, the nature of the Axon Server workload as a database/message hub creates some hurdles. Kubernetes takes care of scheduling and managing workloads which frees up personnel resources for other tasks. But, with the time efficiency using Kubernetes brings, there comes a degree of relinquishing control.

Kubernetes can and will reschedule pods, for example, to make way for higher priority workloads, which is why it is necessary to carefully craft Kubernetes configurations to provide stability and predictability to the deployment on the cluster. Stopping instances of a microservice causes minimal disruption to users, they likely just get an error and hit retry. But stopping a database node without warning can have an impact which is an order of magnitude larger than stopping a microservice and risks making the database unavailable whilst recovery mechanisms kick in. Axon Server has features to handle sudden termination but these are intended for gradual scaling and for disaster situations. Programs can stop all by themselves, for example by a hardware failure, so additional help from Kubernetes is not welcomed that much.

Stateful Sets are the most suitable Kubernetes object to ensure the Axon Server cluster scale in a consistent way. Axon Server also needs access to persistent storage so you need to ensure that the underlying node has provisionable Persistent Volumes. Finally, but not exhaustively, to ensure high availability, you need to ensure Axon Server pods are scheduled on different nodes in different regions or availability zones. For this, you will need to use the Affinity/Anti-affinity configuration.

We should mention here the other options for orchestration tools such as Docker Swarm or Mesos and similar advice applies to these tools.

JARs on Virtual Machines

The main alternative is to take a higher degree of control but with the cost of management and operational overhead. This can be done by running Axon Server on virtual machines which gives you this high level of control and, in theory, allows you to provide a more stable environment for Axon Server to run in. With cloud provisioned virtual machines you can dive into the details which matter most to stateful applications. You can tune the size and speed of storage, network and compute resources that Axon Server has access to as well as giving Axon Server a dedicated host with exclusive access to these resources.

There is no automated orchestrator deciding when and where applications are running. Instead, you are the orchestrator deciding when and which node to stop first for patching or which availability zones to place your nodes.

As with Kubernetes, this approach requires effort on creating a solid configuration for Axon Server to run. Fortunately, there are now a plethora of tools which allow you to describe your Axon Server configuration as code (Infrastructure-as-Code) so that the creation of this infrastructure is automatable with the benefits that come with it.

Conclusion

Balancing the advantages and disadvantages of each approach, we recommend running Axon Server as a JAR on Virtual Machines as our customers report fewer issues with this setup. Running Axon Server on Kubernetes is also possible with some careful configuration and there is a higher chance on encountering hurdles. Wherever you decide to run it, Axon Server paired with Axon Framework is a formidable combination for bootstrapping the development of Domain-Driven Design applications.

Axon Server is a zero-configuration message router and event store for Axon based application. You can download the quick start package here.