Everything I Needed to Know About Observability, I Learned from ‘Bewitched’

Recently, I was asked to write an article on "How to Convince your Boss to Prioritize Observability." You can read it here). As I was pulling it together, one particular sentence sent me down the Wikipedia rabbit hole.

My original draft included a line that mentioned “winning the big _____ account.” My thought was to reference the famous account from the TV show Bewitched that Darren and his partner Larry were always trying to win.

I was sure that in every episode, it was always the same account. When I looked through all the synopses though, I discovered there wasn't just one account. In fact, the writers did a remarkable job of inventing a wide variety of businesses that approached McMann & Tate advertising.

Before I’d done that search, however, I commented on social media about the experience:
https://twitter.com/leonadato/status/1500843766369992707

The response to the tweet caught me off guard, with folks saying they couldn't wait to read about Bewitched and observability.

The problem is, the original idea simply didn't work. An obscure reference to a 70's sitcom wasn't making the article better, so I cut it. But then I felt bad for promising a blog and then failing to deliver it. So here we are. It turns out, there's a heck of a lot of observability insight you can find in old Bewitched episodes — if you look hard enough.

Programming == Magic

It's not only that what we do SEEMS like magic to those untrained in the (sometimes dark) mystical arts of Ruby, JavaScript, or C#. If you look at what witches and warlocks in that old 70's series do — not to mention how they do it — you can see similarities to techniques, methodologies, and approaches like agile, waterfall, and continuous deployment.

Not to mix metaphors (or magical universes) too much, but as The Ancient One told Doctor Stephen Strange,

"The sorcerers of antiquity called the use of this language ‘spells.’ But if that word offends your modern sensibilities, you can call it a program. The source code that shapes reality."

Whether you imagine yourself to be altering the fabric of reality or just the network fabric, it's still a level of responsibility to respect. You may even want a visual reminder of the risk of unintended consequences.

Which means we all must ensure we have a way to understand (i.e., view) the effect our changes are having. It's not enough to run our code through a syntax checker. We need to be able to see how our code is running under production load with real users today, next Thursday, and in perpetuity.

There's No Substitute for Experience

Samantha was certainly the dev… I mean witch… we became most familiar with, but she was by no means the most accomplished or the most powerful. Sam's mother Endorah was widely regarded as one of the most formidable witches we meet, and even humorous characters like Uncle Arther and Aunt Clara were people Sam turned to when she lacked the knowledge or ability to pull off an especially complex bit of magic.

The lesson for us developers is clear, but there are still nuances worth mentioning. Obviously, as developers we have to maintain a level of curiosity, flexibility, and humility such that we can feel comfortable acknowledging when other devs (whether older or younger; with more or fewer years of experience, and so on) have greater fluency, insight, or facility than we do. We should be comfortable acknowledging when we've reached our (current) limit and who we can reach out to for assistance.

Less obviously, we ought to understand how they gained their knowledge. At the heart of any lesson is visibility. Something exposes a fact, truth, or reality to us in a way that informs and changes our view of our work (and sometimes our world) and it gives us both the impetus to change and the understanding of what direction we need to move in.

Sometimes this insight is gained through the school of hard knocks. But more often it's through a tool or technique that lays bare the inner workings of our code, or the platform, or interactions with other systems. Because of this, tools which can provide this insight (i.e., observability) are true "force multipliers," allowing people to BE better now, and improve faster than they otherwise could if they were working without the tool.

The Proof Is in the Punchline

The source of humor in Bewitched often came out of the fact that the developers… I mean witches… lacked insight into the result of their code deploys (spells). Esmeralda accidentally summons Julius Caesar instead of the salad of the same name. Samantha thinks Aunt Clara has turned herself into a cow, but in fact it's a regular cow being used for an advertising campaign. Uncle Arthur means to conjure a cottontail bunny but uses the wrong function call and gets a nightclub hostess instead.

The problem isn't so much the responsibility that comes with wielding great power as it is the responsibility to ensure we can see, measure, and truly understand the impact on the systems around us when we wield that power. Casting a spell without monitoring (or worse still, understanding) the results is bad.

In the sitcom world of Bewitched the result is that "hilarity ensues." In our world as real-life masters of the mystic arts, the results are usually far less amusing.

Which is why writing code without allowing for critical outputs — so called MELT (metrics, events, logs, and traces) — along with the ability to continuously improve our code on the basis of what observability tools tell us, is simply setting us up for experiences that are anything but magical.

The Developers of antiquity might have said this level of insight required a crystal ball. If that idea offends your modern sensibilities, you're free to tell your manager you need a solution that provides observability instead.