Variance: The Heartbeat of Agile Metrics

When teams get their variance right, everything else falls into place.

Variance is a measure of whether teams are doing what they say they are going to do. A team with high variance is over-committing or under-delivering. A team with low variance is delivering on its plans. In this case, stakeholders can feel confident in the team, the team can celebrate at the end of each sprint, and longer-term planning is likely to be accurate.

In this article, we will look at what variance is, how it is calculated, and, most importantly, what Scrum Masters, coaches, and team members can do to get it right.

What Is Variance?

Variance is the difference between story points Committed and story points Done. A rule of thumb is to aim at a variance of no more than 20%.

Points committed are calculated on starting the sprint. The points on all work items in the sprint are summed. Points done are calculated when the sprint ends. All points, attached to items that have been done, are summed.

The two counts can then be displayed on a velocity chart (see below):

Typically, Scrum Masters will do an at-a-glance analysis of variance. If the two bars are close together, all is good. If Committed is, say, double the Done, then some action is required. So in the example above, we can see that variance looks too high in the first three sprints, but much better in the following three. See the Appendix for how to calculate variance.

Why Is Variance So Important?

Variance is so important for Agile teams. Here is why:

1. Barometer of Team Health

If a team has low variance — i.e., it is doing what it is committing to do — it is an indicator that many other things are going well. For example:

1. Backlog refinement and estimation: A low variance suggests the team is effectively managing the backlog and doing accurate estimation.
2. Sprint planning: If the team has low variance, they are doing what they are committing to do. So during sprint planning, they must be taking in a manageable amount of work into the upcoming sprint. They must be managing their capacity.
3. Definition of Ready (DoR): Low variance suggests that the team has a solid definition of DoR and is applying it to all items put into a sprint (more on DoR below).

Conversely, high variance suggests there are issues with some of the above.

2. Impact on Other Metrics

Burndowns

If a sprint has low variance, it means the team has done what it planned to do. Of course, it won’t determine the shape of the burndown line, but it does mean the line should end up close to zero.

Cycle Times

I am thinking here about the average time a work item takes to get done; i.e., from the time it gets put into a sprint to the time it is put into a done state. If variance is low, it means that all or most of the work items in a sprint are getting done by the end of it. So, cycle times will be around the same as the duration of the sprint. This is normally the target for cycle times.

3. Longer Term Planning

Increasingly across the industry, organizations are making efforts to align the work of the scrum teams to longer-term plans. 

This can be done in a variety of ways:

1. Product goals: To align the goals to long-term plans, they can be set to be time-bound and specific.
2. PI planning: Organizations doing some form of scaled agile are often using the idea of PI (Product Increment) Planning. This is where the team and stakeholders (who are working on the same product) meet periodically (typically quarterly) to align to a shared vision, plan deliverables, identify risks and dependencies, etc.
3. Roadmaps: Some organizations are aligning scrum teams to roadmaps. While this may feel a bit non-agile, I would argue that as long as these roadmaps are based on empirical data, coming from the squads, it is most likely beneficial for the organization.

No matter what form of long-term planning is being used, teams with low variance will be able to accurately plan for the future.

Four Ways to Improve Variance

1. Definition of Ready (DoR)

When a team has high variance, they are typically taking items into the sprint that are not ready.

In this short video by Jeff Sutherland, he walks us through some of the key aspects of DoR. They include:

• Immediately actionable: The team should be able to start work on this item on Day 1 of the sprint. If they are waiting on some input from a stakeholder or a deliverable from another team, it is not ready and does not belong in a sprint.
• Doable in one sprint: If it is too big to get Done in one sprint, it is not ready and should be broken down into smaller items.
• Understood: Discussions with the PO and the stakeholder should have already happened. The team should know exactly what needs to get done to deliver the item. Otherwise, it is not ready.

When teams take in items that are not ready they are likely to get blocked, delayed, and put on hold. Variance will be impacted. The best practice is to rigorously check all work items against DoR during sprint planning.

2. Push Back on Pressure From Stakeholders

As deadlines loom and pressure builds, Agile teams are sometimes pressured into taking on too much work in a sprint. They will often justify their actions with heroic phrases: “We will pull out all the stops," “We will go the extra mile," etc.

Sadly, it doesn’t often happen. The team cannot deliver on an unrealistic workload and has to admit to stakeholders that they are not going to get what was promised – an uncomfortable situation for all concerned.

There are many concerns about over-committing:  

Stability

In Agile, we are aiming for stability. One of the principles of the Agile Manifesto states:

"Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely."

The False Promise of Longer Working Hours

There have been numerous studies that have suggested an increase in working hours actually reduces productivity. Encouraging people to spend long hours working actually decreases the work they get done.

The Downward Spiral

Typically, working long hours sets up a downward spiral: work/life balance is impacted, physical and mental health are impacted, people get demotivated and frustrated with their work, productivity goes down, quality goes down and people start looking for other jobs. High staff turnover always has a major negative impact on productivity.

One solution lies in the fundamentals of Agile. One of the pillars of scrum is transparency. Teams must make their work visible. They must be open and honest about what they can do, and also what they can’t do.  

One of the scrum values is courage. It takes some courage to say to managers: “I am sorry, but we don’t have the capacity to complete these items this sprint." But it is far better than committing to doing something and then failing to deliver on it. 

3. Avoid Overly Optimistic Estimation

There is a tendency for team members to be overly optimistic when estimating and deciding how much they can get done in a sprint. There are several root causes: 

1. Happy Day Scenario: They are only considering the Happy Day Scenario. They assume that everything will go according to plan and they will not face any issues with the work.
2. Entirety of work item: They are not considering the entirety of the work item. Most work items consist of doing – reviewing/testing – re-working – re-reviewing/acceptance. There is a tendency to only consider the doing bit.
3. Historical data: They are not considering historical data and looking at how long it took to do similar items in previous sprints.

Scrum Masters have a role to play here. They can coach the team on estimation, helping them to take a holistic approach and consider the overall sizing of the item, not just the doing bit. They can bring in historical data to inspect how long similar items have taken to do in the past. 

4. Do Capacity Planning

It is dangerous to assume that all team members will be working on all the days of the sprint. In distributed teams, there may be public holidays in various locations, and team members may be taking personal time off. It is important to track this data as it will impact the capacity of the team in the upcoming sprint.

This was a mistake I made when I first became a Scrum Master. We would meticulously do our sprint planning basing the committed story points on the velocity of previous sprints. Then midway through the sprint, I would realize that several team members had gone on vacation and forgotten to tell me or there were public holidays in other countries that I didn’t know about. 

I very quickly implemented a team calendar (I used the one in Confluence, but there are many tools that will do the job). I regularly reminded the team members to put in personal leave and any public holidays in their country or region. During sprint planning, one of the first activities was to review the calendar and determine the capacity of the upcoming sprint.

Conclusions

We have seen that a team with low variance is most likely a high-performing team who delivers on their plans. And we have looked at several techniques teams can use to reduce their variance. As variance decreases, stakeholders gain confidence in the team; they know they are likely to get what has been planned. Best of all, at the end of each sprint, the team can celebrate delivering what they set out to deliver.

Appendix: Variance Calculation

Variance can be calculated using the following formula:

Variance = (Committed – Done) x 100/Committed

In the above example in Sprint 1:

Committed = 56

Done = 20

Variance = (56 – 20) x 100/56  = 64%

As mentioned above, a good rule of thumb is to aim to keep variance below 20%.