Open Sourcing Geppetto, an Automation Framework for Distributed Systems

After graduating college and entering the workforce, I began to see just how ubiquitous and useful open source software is — from implementations of C++ and Java to the many software libraries to even Ansible and OpenStack for virtual machine management. When the CEO of my company approached me and mentioned that we could open source our automation framework for distributed systems, I was excited to contribute back to the community.

At Datos IO, we developed an automation framework we call Geppetto to help manage and test RecoverX, one of our software products. RecoverX is a database versioning and backup solution for cutting-edge distributed databases. In order to handle the scalable demands of distributed databases, RecoverX is itself distributed. Perhaps more so than any normal piece of software, RecoverX needs to be tested thoroughly because, after all, RecoverX is protecting customers' data in a nondeterministic distributed environment.

"Developers lie at the heart of a software project. They implement the experiences that customers will rave about at release. Too often, developers don't get to own the entirety of their feature life cycle. Traditional test methodology holds that testing is a separate process out of step with the developer. Testing really is a collaborative process between the developer, the team, and the customers. Newer methodologies like test-driven development promote more ownership for developers and collaboration with the team. Developers that own and prove the quality of a feature have fewer bugs, fewer round trips with QA, and are faster to market. A structured series of tests owned by the developer and test teams help protect code from incoming bugs. Developers that write tests are also more likely to keep their quality up."— Atlassian

At Datos IO, we know that developers are an integral part of testing. But we also know their time is incredibly valuable. After all, we are a small startup with fewer than 30 employees taking on a giant enterprise market.

We knew right away that we had to automate as much as possible, but we also knew that there was more to it than that. We needed to make automating powerful, flexible, and, most importantly, easy. Being easy to use is so important because it promotes software uptake. If it's easy to use the automation software, the developers are much more likely to use it.

Also, consider the cost benefit. Every minute or hour saved in learning a new piece of software is time that could be spent developing new features or products. So you not only have the cost of the developers (multiplied by all of those hours multiplied by the number of engineers you have that you save in cost) but you're also increasing your product value with new features that are much quicker, thus increasing revenues.

Geppetto Overview

Geppetto is developed in Python, an easy-to-use and well-supported cross-platform language. We took inspiration from the structure of Python's unit tests for its simplicity and familiarity that developers have with it. Our thinking was if we mirror the Python unit test structure a developer should be able to learn how to implement a Geppetto test quickly.

For example, a simple "hello world" test can be implemented in just a few lines.

class TestRun(Geppetto):
    def run(self):
        report(“Hello World!”)

If you want to extend the test to include setting up a distributed database cluster, like Cassandra for example, Geppetto can work with a configuration file that stores IP addresses, usernames, passwords, and other configuration settings. This allows setting up a Cassandra cluster in a Geppetto test to be incredibly easy and implemented in just two extra lines

class TestRun(Geppetto):
    def run(self):
        cassandra = Cassandra(config_dictionary)
        cassandra.install()

We now have a Cassandra cluster set up. But perhaps we want to take this a step further and test one of many different failure scenarios for Cassandra. Let's extend our test a bit more.

class TestRun(Geppetto):
    def run(self):
        cassandra = Cassandra(config_dictionary)
        cassandra.install()
        cassandra.failure.random_db_failure()

In just five simple lines of code, we've created a Cassandra-distributed database and inserted failures into the Cassandra database. And, of course, with a combination of different available functions, parameters, and configuration files, and the full availability of Python libraries, such as threading and multiprocessing, we now also have not just an easy tool to use, but a very capable and flexible one as well.

Key Design Principles

Geppetto was designed with some key principles in mind:

• Modularity: Extendable and easy to add functionality. Future proof.
• Flexible and Powerful: Able to handle any kind of testing requirements.
• Ease of Use: Abstract away all the hard work. Engineers should spend more time focused on testing and development, not on how to test.

What Can Be Automated

Geppetto can automate either a single test or hundreds or even thousands of tests. A test can run for milliseconds or even months and can be as simple as a simple “Hello World” or as complex as setting up many different versions of different types of distributed systems, perhaps populating them with terabytes of data and doing complex things with them. In fact, I even use Geppetto as a simple setup tool to turn on and off firewalls and do monitoring of clusters for things like CPU and memory usage. The sky is the limit.

In the Wild

We are now releasing a version of Geppetto to open source, and we will be adding in many modules over the coming months. We are actively looking for contributors and hope that individuals in the open source community will find it of some use.

Download

You can download Geppetto here.