10 Deep DevOps Thoughts From Chef’s Jez Humble

The IEEE’s DevOps Unleashed symposium in Silicon Valley last week promised “expert advice on how to innovate faster by accelerating software delivery across the enterprise.” It pretty much delivered, with plenty of great insights into the theory and practice of DevOps in the enterprise. But for many attendees, the highlight was the Enterprise DevOps presentation from Jez Humble, vice president at Chef and co-author of a pair of books on the topics, Lean Enterprise: How High Performance Organizations Innovate at Scale and Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.

Jez was engaging, knowledgeable, and funny, and the audience of several hundred developers, ops folks, and business leaders paid rapt attention. He touched on many crucial topics, and you can see his slides here, but these 10 observations should give you an idea of his perspective, along with some valuable best practices:

1. DevOps is not a goal, but a never-ending process of continual improvement.

The idea is to create a culture of everyone wanting to continuously get better.

2. IT performance metrics are not the same as IT performance predictors.

These are the top four metrics of IT performance, Jez said, according to the 2014 Puppet Labs State of DevOps Report:

1. Lead time for changes
2. Release frequency
3. Time to restore service
4. Change fail rate

And these are the top five predictors of IT performance by those metrics:

1. Peer-reviewed change approval process
2. Version control everything
3. Proactive monitoring
4. High-trust organizational culture
5. A win-win relationship between dev and ops

Jez Humble, VP of Chef

3. Metrics must evolve.

DevOps shops should avoid using the same metrics all the time, Jez said. “You have to evolve your metrics,” he explained, because “every time you measure something, it changes behavior.” Just as important, he added, the people who are being measured have to buy in to those metrics or they’ll figure out a way to game the system. Too often, management uses metrics as an offensive weapon, and “that never works.” People will do what they’re told, but they’ll find a way to make their metrics look good whether or not they actually help the organization meet its goals.

See also: Taking a Closer Look at Data Culture in Software Development [Survey Report]

4. You don’t have to trade throughput for stability.

Jez noted that the 2014 Puppet Labs State of DevOps Report separated the key indicators for IT quality into two flavors: throughput (including lead time for changes and release frequency) and stability (including time to restore service and change fail rate).

According to Jez, this is not a zero sum game where you have to trade throughput for stability. “If you’re doing it right,” he said, “it’s quite the contrary.” In fact, high-performing IT organizations did much better on both throughput and stability, while low-performing shops did worse in both areas.

5. External change approval processes don’t work.

Jez described external change approval processes as “Risk management theater.” Because external code reviewers don’t know the code base, these kinds of reviews slow throughput and don’t help stability. Jez asked how outside reviewers are expected to review thousands of lines of code by hundreds of programmers and predict its impact on the production environment. “It can’t possibly work, which is why it doesn’t work,” he said.

A peer review process, however, speeds throughput and doesn’t hurt stability, Jez said, because developers work on the code base and know it inside and out.

6. Having an emergency change process is a terrible idea.

Typically, emergency change processes leave out testing. That’s a terrible idea, Jez said. You already have problems or you wouldn’t be invoking the emergency process to fix it, but making changes without testing could make things worse instead of better. “You want to use the normal change process for emergency situations,” Jez said. “That’s how you know you’ve got it right.”

7. Continuous delivery requires developers to check their code into the trunk daily.

“DevOps is a practice, not a tool,” Jez said. The key to continuous delivery is by architecting for it. All code must be independently deployable, and should be tested before submitting it to the trunk, not to some enormously complex/expensive integration environment. “This doesn’t require a tool,” he said, “just the mindset that software is always in a working state.” And it doesn’t hurt for developers to break up large features into small, incremental changes.

On the other hand, if everyone works on feature branches, they have to integrate them not just with trunk, but with each other. And that requires very sophisticated test automation.

8. Leaving broken code in the trunk is selfish.

If you can’t make it work in a few minutes, Jez said, back out the change. Any change needs to be in version control, and if it breaks, developers need to fix the problem straightaway and test the new code.

9. Software testers are not responsible for quality.

Everyone is responsible for software quality, Jez said. Testers are there to make quality apparent so it can be worked on, not to add quality to the project. This means that testing is not something to do after development is complete, but a core part of the development process—all the time at every stage. Remember, these tests are designed to prove only that the code is not deployable, not that it is ready to go. “If it makes it through all these tests and you’re still feeling even a bit nervous about deploying,” Jez said, “that means your tests aren’t good enough.”

10. Less is more, more or less.

Research shows that even well-designed and executed projects developed to improve a key metric succeeded in doing so only about a third of the time. Another third of the time they didn’t help significantly, and the remaining third of the time they actually made things worse! For Jez, that metric suggests that “doing less” is often the best strategy, especially when implementing DevOps in the enterprise.

“Users don’t know what they want. Users know what they don’t want once you’ve built it for them.” Business objectives are what the customers demand.

Bonus point: Bi-modal IT is a massive reductionist oversimplification.
IT should treat every project or process according to its own characteristics, Jez said. And DevOps can help you do that.