Engineering Productivity: What Not to Do

Motivated by some recent M&A news and general productivity pressures in time of tight budgets, we present some anti-patterns to the use of engineering metrics and give an overview of how to use metrics for productivity insights instead. First, let us first start with what to avoid: 

Anti-Pattern to Engineering Metrics

Lines of code written: While lines of code can be a proxy metric for how much code you have to maintain (and therefore costs to incur), it has been proven to be a terrible metric for productivity. This shouldn’t need much explanation, but: Output does not necessarily equate to the outcome. Different programming languages have different verbosity, and e.g., just including some open-source packages or pasting in some code from Stack Overflow does not make you more productive. And, of course, solving hard problems often does not require a lot of code but a lot of thinking, exploration, and collaboration.

Complexity or salience of code: Complex code is not good code. Someone else has to read, understand, and maintain it. Salient code (let’s interpret this as “most notable”) can be worth noting, but that does not make you a good team player or show any consistency. One key point here: Software development is a team sport. However, every company has some rock stars who are able to do things that others cannot, but that does not mean you only want rock stars who cannot work with others. This is especially true if you have more than on star with similar traits.

Number of PRs/Reviews: Now, this is a bit more on the grey side. Of course, if you can churn out a lot of pull requests (PRs), this typically means it is work associated with some feature spec (at least one would hope so). This means it is at least going in the direction that is perceived as customer value. But on their own, PRs are again a proxy of the amount of work that requires maintenance and a metric that is prone to gaming. Sure, you can do a lot of small PRs and “clog up” the review and merge pipeline, or do a lot of cursory reviews (LGTM), or worth add confusing and useless comments. So, adding value to some customer features is good, but for the sake of showing activity, it is not.

Bug fixes: Someone fixes a zillion bugs? This can be great, but two questions come to mind: Why are there so many bugs in the first place? And: do all of these “bugs” need fixing? The unfortunate truth is that no software is perfect, and there is always way more to do than that there is time for.

Number of hours worked: This shouldn’t need much explanation either, but hours do not equate to time well spent. Sure, if you only work 1h a day, you must be really exceptional (and not hardcore) to produce continuously high value. But if you work 14h, it does not mean you are at your most productive. Especially running out of steam/coffee. All good things need the right effort without going to extremes. Sometimes this might be more coding, and sometimes more exploration/thinking/discussing.

The Value of Data-Driven Engineering

Now, we are the last ones not to promote some metrics (ahem ….), but metrics are not the goal; insights into what is going on in the organization and continuous improvement loops are.

Firstly, metrics are almost always poor to manage individuals. If you need metrics to understand if an employee gives you the value you are expecting, then you are already in trouble. We understand that large organizations need some sense of objectivity, but using any of the above is probably poor. Also, metrics in themselves do not solve principal issues. This brings us to the next point:

Metrics are of value to highlight trends, anomalies, imbalances, and bottlenecks. This is, in particular, true for processes, workflows, and team/group aggregates. It would be unthinkable to run your ops without watching some numbers, and the same for sales or marketing. But it is always intended to improve and streamline processes and create less disruption. As such, metrics can provide great signals to manage organizational engineering productivity and remove friction in the system. 

For development teams, it makes sense to measure some numbers and trends. For instance: Where in the development process do we get stuck or waste time? Does our build system hold us back? Are we overloaded with reviews? Do we have QA issue? Do our sprints get re-scoped all the time in mid-air? Are there too many context switches? All these things are essential, especially for good managers to assist their teams. Managers should remove roadblocks and distractions, not micromanage lines of code.

As for trends, there are some indicators of whether teams are “in the flow” or not. How are the features/PRs tracking over time? How much work do we usually get done (feature tickets, even PRs as a proxy)? Do we follow the process (reviews and approve those)? Do our security checks run? How does our build reliability perform? The question then is: Do we see spikes, anomalies, or unhealthy trends? These indicate issues with infrastructure, processes, and team health. Those warrant fixing and, in turn, create happier and more productive teams.

Reducing Friction, Not Micro-Managing 

The key to using metrics is to gain insights and confidence into processes, teams, and workflows that are otherwise hard to obtain. The engineering productivity objective should be to minimize any friction in the software delivery pipeline. This means the steps from planning a feature to delivering it to the customer and the reaction time to customer feedback/requests should be quick, smooth, and reliable. Any process or principle bottlenecks significantly outweigh a single contributor's shortcomings and should therefore be the major focus.

Modern engineering leaders focus on the 4 Key Metrics to structure their engineering measurements to get better visibility into their teams and delivery processes. These metrics are:

1. Velocity of delivery (how fast)
2. Throughput of delivery (how much)
3. Quality of delivery (how good)
4. Impediments to delivery (how risky)  

Recently, a new wave of engineering intelligence platforms supports teams and their engineering managers to get the required visibility and insights to create such healthy and productive organizations. By shifting the focus away from individual performance metrics to reducing friction towards productivity goals, companies have achieved double-digit percentages of cost savings after a short period of time. We will dive deeper into this in a future article.