The Curse of Simplicity: The Simplest Doesn’t Mean the Least Sophisticated

It is often said that software developers should create simple solutions to the problems that they are presented with. However, coming up with a simple solution is not always easy, as it requires time, experience, and a good approach. And to make matters worse, a simple solution in many ways will not impress your co-workers or give your resume a boost.

Ironically, the quest for simplicity in software development is often a complex journey. A developer must navigate through a labyrinth of technical constraints, user requirements, and evolving technological landscapes. The catch-22 is palpable: while a simple solution is desirable, it is not easily attained nor universally appreciated. In the competitiveness of software development, where complexity often disguises itself as sophistication, simple solutions may not always resonate with the awe and admiration they deserve. They may go unnoticed in a culture that frequently equates complexity with competence.

Furthermore, the pursuit of simplicity can sometimes be a thankless endeavor. In an environment where complex designs and elaborate architectures are often celebrated, a minimalist approach might not captivate colleagues or stand out in a portfolio. This dichotomy presents a unique challenge for software developers who want to balance the art of simplicity with the practicalities of career advancement and peer recognition.

As we get closer to the point of this discussion, I will share my personal experiences in grappling with the "curse of simplicity." These experiences shed light on the nuanced realities of being a software developer committed to simplicity in a world that often rewards complexity.

The Story

Several years ago, I was part of a Brazilian startup confronted with a daunting issue. The accounting report crucial for tax payment to São Paulo's city administration had been rendered dysfunctional due to numerous changes in its data sources. These modifications stemmed from shifts in payment structures with the company's partners. The situation escalated when the sole analyst responsible for manually generating the report went on vacation, leaving the organization vulnerable to substantial fines from the city hall. To solve the problem, the company’s CFO called a small committee to forward a solution. 

In advocating for a resolution, I argued against revisiting the complex, defunct legacy solution and proposed a simpler approach. I was convinced that we needed "one big table" with all the columns necessary for the report and that each row should have the granularity of a transaction. This way, the report could be generated by simply flattening the data in a simple query. Loading the data into this table should be done by a simple, secure, and replicable process.

My team concurred with my initial proposal and embarked on its implementation, following two fundamental principles: 

1. The solution had to be altruistic and crafted for others to utilize and maintain.
2. It had to be code-centric, with automated deployment and code reviews through Pull Requests (PR).

We selected Python as our programming language due to its familiarity with the data analysis team and its reputation for being easy to master. In our tool exploration, we came across Airflow, which had been gaining popularity even before its version 1.0 release. Airflow employs DAGs (Direct Acyclic Graphs) to construct workflows, where each step is executed via what is termed "operators." Our team developed two straightforward operators: one for transferring data between tables in different databases, and another for schema migration. This approach allowed for local testing of DAG changes, with the deployment process encompassing Pull Requests followed by a CI/CD pipeline that deployed changes to production. The schema migration operator bore a close resemblance to the implementation in Ruby on Rails migration. We hosted Airflow on AWS Elastic Beanstalk, and Jenkins was employed for the deployment pipeline. During this period, Metabase was already operational for querying databases.

Within a span of two to three weeks, our solution was up and running. The so-called "one big table," effectively provided the accounting report. It was user-friendly and, most crucially, comprehensible to everyone involved. The data analysis team, thrilled by the architecture, began adopting this infrastructure for all their reporting needs. A year down the line, the landscape had transformed significantly, with dozens of DAGs in place, hundreds of reporting tables created, and thousands of schema migration files in existence.

Synopsis of the Solution

In essence, our simple solution might not have seemed fancy, but it was super effective. It allowed the data analysis team to generate reports more quickly and easily, and it saved the company money on fines. 

The concept of the "curse of simplicity" in software development is a paradoxical phenomenon. It suggests that solutions that appear simple on the surface are often undervalued, especially when compared to their more complex counterparts, which I like to refer to as "complex megazords." This journey of developing a straightforward yet effective solution was an eye-opener for me, and it altered my perspective on the nature of simplicity in problem-solving.

There's a common misconception that simple equates to easy. However, the reality is quite the contrary. In reality, as demonstrated by the example I have provided, crafting a solution that is both simple and effective requires a deep understanding of the problem, a sophisticated level of knowledge, and a wealth of experience. It's about distilling complex ideas and processes into their most essential form without losing their effectiveness. 

What I've come to realize is that simple solutions, though they may seem less impressive at first glance, are often superior. Their simplicity makes them more accessible and easier to understand, maintain, and use. This accessibility is crucial in a world where technology is rapidly evolving and there is a need for user-friendly, maintainable solutions.