Why Agile Engineering Practices in Software Development Are Essential to Achieve Agility

In the Oxford Dictionary, the word agility is defined as "the ability to move quickly and easily." It is, therefore, understandable that many people relate agility to speed. Which I think is unfortunate. I much prefer the description from Sheppard and Young, two academics in the field of sports science, who proposed a new definition of agility within the sports science community as "a rapid whole-body movement with a change of velocity or direction in response to a stimulus" [1]. 

The term “agility” is often used to describe “a change of direction of speed.” However, there is a distinct difference between “agility” and “a change of direction of speed.” Agility involves the ability to react in unpredictable environments. Change of direction of speed, on the other hand, focuses purely on maintaining speed as the direction of travel is changed. Maintaining a speed while changing direction is usually only possible when each change of direction of travel is known in advance. 

Using a sports analogy as an example, in soccer, we can say that a defender’s reaction to an attacker’s sudden movement is an agility-based movement. The defender has to react based on what the attacker is doing. Compare this to an athlete running in a zig-zag through a course of pre-positioned cones, and the reactive component is missing. There are no impulsive or unpredictable events happening. The athlete is trying to maintain speed while changing direction.

Often, when leaders of organizations want to adopt Agile, they do so for reasons such as “to deliver faster.” In this case, they are thinking of agile as a way to enable a change of direction of speed like the athlete, and not in the sense of agility needed by the soccer defender when faced with the attacker. This may explain why agile ways of working do not always live up to expectations, even though more and more companies are adopting it.

Sticking with the sports analogy, the athlete running through the cones tries to reach each one as quickly as possible and then runs in the direction of the next until the end of the course. This works as a metaphor for defining the scope of a project and has teams work in short iterations in which they deliver each planned feature as quickly as possible and then move on to the next. This may be fine in a predictable environment, where the plan does not need to change, where requirements stay fixed, where the market stays the same, and where customer behaviors are well understood and set in stone.

In many environments, however, change is a constant: customer expectations and behavior, market trends, the actions of competitors, and more. These are the VUCA environments (volatile, uncertain, complex, and ambiguous) where there is a need to react or, to put it another way, where agility is needed. Frameworks such as Scrum are meant to support agility. Sprints are short planning horizons, and the artifacts and events in Scrum are there to provide greater transparency and opportunities to inspect and adapt based on early feedback, changes in conditions, and new information. It gives an opportunity to pivot and react in a timely manner. However, Scrum is unfortunately often misunderstood as a mechanism to deliver with speed.

Focusing only on speed and delivery and not investing in the practices that enable true agility is likely to actually slow things down in the long run. When the focus is only on speed, it becomes harder to maintain that speed, let alone to increase it, and any semblance of agility is a fantasy. Let me talk about a pattern that I see again and again as an example.

Company A has a goal to build an e-commerce site through which they can sell their goods. Their first slice of functionality is delivered in a 1-month Sprint and consists of a static catalog page in which the company can upload its products with a short description. The first delivery is received by happy and excited stakeholders who are hungry for more. The team keeps on building, and the product keeps growing.

Stakeholders make more requests, and the team works harder to keep up and deliver at the pace that stakeholders have come to expect from them. The team does not have time to invest in improving their practices. Manual regression testing becomes a bigger and bigger burden and is more challenging to complete within a Sprint. The codebase becomes more complex and brittle. The more bloated the product becomes, the more the team struggles to deliver at the same pace. To try to meet expectations, the team begins to cut corners. They end up carrying testing work over from one Sprint to the next. And there is no time for validating if what is being delivered actually produces value. This is just as well, as no analytics have been set up anyway.

In the meantime, whilst the team is so busy trying to build new features and carrying out manual integration and regression testing, they do not have time either to look at improvements to their original build pipeline or to build in any automation. A release to production involves several hours of downtime, so this has to be done overnight and manually. To make matters worse, the market has been changing. The sales team has made deals with new suppliers, but this means further customizations to the site are needed for their products. Finally, the company has pushed for the platform to be available in different timezones, so the downtime for a release is a big problem and must be minimized, so they are only allowed to happen once every six months.

Progress comes to a standstill. The product is riddled with technical debt. The team has lost the ability to get early feedback and the ability to react to what their attackers are doing, i.e., customers' changing needs, competitors’ actions, changing market conditions, etc.

Just implementing the mechanics of a framework like Scrum does not ensure agility and does not automatically lead to a more beneficial way of working. The Agile Manifesto includes principles such as “continuous delivery of valuable software,” “continuous attention to technical excellence,” and “at regular intervals, the team reflects on how to become more effective.” Following these principles is a greater enabler of agility than following the Scrum framework alone. One effective way of enabling greater agility is to complement something like Scrum with agile engineering practices to get those expected benefits that organizations are looking for with agile.

Over the years, I have encountered many Agile adoptions at companies where a lot of passion, energy, focus, and budget went into training and coaching people in implementing certain frameworks such as Scrum. However, what I do not encounter so much are companies spending the same amount of passion, energy, focus, and budget into implementing good Agile engineering practices such as Pair Programming, Test Driven Development, Continuous Integration, and Continuous Delivery. When challenged on this, responses are typically something like “We don’t have time for that now," or  “Let’s just deliver this next release, and we’ll look at doing it at a later date when we have time." And, of course, that time usually never arrives.

Now, of course, something like Scrum can be used without using any Agile engineering practices. After all, Scrum is just a framework. However, without good engineering practices and without striving for technical excellence, a Scrum team developing software will only get so far. Agile engineering practices are essential to achieve agility as they allow the shortening of validation cycles and get early feedback. For example, validation and feedback in real-time on quality when pairing, or that comes with proper Continuous Integration being in place. 

Many of the Agile engineering practices are viewed as daunting and expensive to implement, and doing so will get in the way of delivery. However, I would argue that investing in engineering practices that help to build quality or allow tasks to be automated, for example, enables sustainable development in the long run. While investing in Agile engineering practices may seem to slow things down in the short term, the aim is to be able to maintain or actually increase speed into the future while still retaining the ability to pivot. 

To me, it is an obvious choice to invest in implementing Agile engineering practices, but surprisingly, many companies do not. Instead, they choose to sacrifice long-term sustainability and quality for short-term speed.  

Creating a shared understanding of the challenges that teams face and the trade-offs for short-term speed without good engineering practices in place versus the problems that can arise without them can help to start a conversation. It is important everyone, including developers and a team’s stakeholders, understands the importance of investing in good Agile engineering practices and the impact on agility if you don’t do it. These investments can also be thought of as experiments — trying out or investigating a certain practice for a few iterations to see how it might help can be a way to get started and make it less daunting.

Either way, it is questionable that an Agile process without underlying agile practices applied can be agile at all. For sustainability, robustness, quality, customer service, fitness for purpose, and true agility in software development teams, it is important for continuous investment in agile engineering practices.

 [1]   J. M. Sheppard & W. B. Young (2006) Agility literature review: Classifications, training and testing, Journal of Sports Sciences, 24:9, 919-932, DOI: 10.1080/02640410500457109