Essential Complexity Is the Developer's Unique Selling Point

In my previous post, I highlighted the difference between efficiency and effectiveness and how it maps to artificial versus human intelligence. Doing things fast and with minimum waste is the domain of deterministic algorithms. But to know when we’re building the right thing (effectiveness) is our domain. It’s a slippery and subjective challenge, tied up with the confusing reality of trying to make human existence more comfortable with the help of software.

Today I want to talk about essential complexity. A fully autonomous AI programmer would need to be told exactly what we want, and why, or it should be sufficiently attuned to our values to fill in the gaps. Sadly, we cannot trust AI yet to reliably connect the dots without human help and corrections. It’s not like telling an autonomous vehicle where you want to go. That has a very simple goal – and we’re nowhere near a failsafe implementation.

Essential complexity is about “debugging the specification,” figuring out what we, the people, need and why we need it. Accidental complexity is a consequence of the alternatives we choose to implement these ideas. Frederick Brooks’ enduring distinction between essential and accidental complexity is analogous to the realms of human vs. machine intelligence similar to the effectiveness/efficiency distinction of the previous post.

Since fully autonomous software production by businesspersons could only work if they state exactly and unambiguously what they want, developers smugly conclude that their jobs are safe. I’m not so sure that such perfect specs are a necessary condition. I mean, they aren’t now. Who postpones coding until they have complete, unambiguous, and final specifications? Programming means fleshing out the specification in your IDE from a sufficiently clear roadmap, filling in the details as you go along. It’s not mere implementation; it’s laying the bricks for the first floor while you’re still tweaking the drawing for the roof. It seems inefficient, but it turns out we can’t imagine our dream house perfectly unless we’re halfway done building it, at least when it comes to making software.

AI is already very qualified to deal with much of the accidental complexity you encounter on the way. We should use it as much as we can. I know I devoted three articles to the Java OCP 17 exam (link here for Part 1, Part 2, and Part 3), but I believe (and hope) that rote knowledge of arcane details will go the way of the dodo. AI takes care of idiomatic usage, it can enforce clean code, good naming conventions, and even write source documentation. And it will get better and better at it. It can even do full-blown migrations of legacy code to new language and framework versions. I’m all for it. Migrating a Java 4 EJB2 behemoth to Spring Boot 3 microservices by hand is not my idea of fun.

If in five years’ time, the state of the art in code assistance still leaves you unimpressed while writing code, it’s probably not because of some accidental complexity the machine can’t handle. It’s most likely the essential complexity it can’t deal with. If your mortgage calculator outputs a 45.4% mortgage interest rate and the co-pilot won’t warn you that you probably confused a decimal point, it’s because it has never bought a house itself and won’t notice that the figure is an order of magnitude too steep.

Essential complexity can be expressed in any medium; it needn’t be computer code. Once you know exactly how something should work, most coding challenges become easy by comparison, provided you are competent in your language of choice. So, we break down complicated domains into manageable chunks and we grow the product, improving and expanding it with each iteration. That doesn’t always work. Sometimes the essential complexity cannot be reduced, and you need a stroke of genius to make progress. 

Take for example asymmetric key exchange, a tantalizing problem that tormented the greatest mathematical minds for decades, if not centuries. Alice and Bob can communicate using an uncrackable encryption key, but if they don’t know that Eve has intercepted it, everything is in the open. If only we could have a pair of keys, such that you can encrypt a message with key A but can only decrypt it with key B and no practical way to deduce one key from the other. If you then give out one part of the key to everybody and protect the other part of your life, you have solved the key exchange.

It's simple enough to state where you want to arrive but it’s hardly a specification from which to start coding. It’s not even a programming task. It’s the quest for inventing an algorithm that may not even be possible. In Scrum Poker you would draw the infinity card. The algorithms that Whitfield Diffie and Martin Hellman ultimately devised fit on the proverbial napkin. Translating it to code would be trivial by comparison. But they could never have arrived at the solution incrementally behind a keyboard. Or read about the fascinating story of cracking the Enigma cipher by the team at Bletchley Park. An even more daunting task because there was literally a war to win. 

You cannot make a masterpiece to order, nor in the art of science. If we knew what made a good song, we could replicate the process, if not using software, then at least by some formulaic method. But that doesn’t produce classics. Creativity is a hit-and-miss process and few artists consistently produce works of genius. There’s no reason why we should expect great AI advances in that department. But we can expect better tooling to get the creative juices flowing. Songwriters use a rhyming dictionary and thesaurus in search of inspiration. That’s not cheating.

Fortunately, unless you’re working at a university or research institute, enterprise software development and maintenance isn’t about solving centuries-old math conundrums. However, you should ponder more deeply what we want and need, instead of learning a cool new framework or getting another AWS certificate. Uncovering the essential complexity is not just the job of business analysts in the organization. I can’t wait for next-generation tooling to help us grapple with it, because that would be a genuine copilot instead of an autopilot.