Data DevOps - What and Why

I had the opportunity to catch up with Derek Hutson, President and CEO of Datical to get his thoughts on how to handle the ever-growing flood of data and how to integrate data and databases into software and application development.

What is Data DevOps?

We define Data DevOps as the process of having data and internal systems, such as a database, that allow for testing at the speed, quality, and security required for a modern DevOps environment. 

Why do we need Data DevOps?

There are so many different and disparate sources of data, and the rate at which data is being produced is close to exponential. That trend is not going to slow down. When we were releasing once a quarter, there was time for the data to catch up; there was time to do data integration. However, today, if you want to release every week, you have to have data DevOps.

You may think, "I'm not doing it now. Why do I need it?" You're missing out on opportunities and increasing your cost. Evaluate the rate and pace of innovation and the rate and pace of your agility to be innovative. Innovation and DevOps deployment has increased application release cycles, as well as the speed at which we can provision at the platform layer with the cloud. The slowest part of that is often the data layer. We need to accelerate the data layer because of the pace at which we're releasing everything else.

The other thing I would say, as a user, is that an application is meaningless to me without my own data. Think about when you download a new app to your smartphone. It's useless until you connect your data; the user-experience is directly tied to data and would not be the same without it.

Often, we don't focus on data. That's why I think we've got to be focused on Data DevOps. What we really want is a bespoke experience for ourselves.  Services, applications, and user experiences are much better with our data. Enterprises have to be adept at delivering data that their users want and expect since the bar is going up every single day.

What problems do Data DevOps address?

I would say speed, quality, and lack of errors. In DevOps, there is a great statistic about the number of unplanned outages that no longer occur because of maintenance that you do.  There's a lot of maintenance required at the data layer, the database layer, and much more so at the app, then at the application layer, just that lack of errors equates to better quality.

But there’s another important consideration as well. That's security. There are times when we'll go into an organization and they will be trying to do DevOps for their database, and they will have compromised their security in some way or another. They're going so fast they are not considering all the security implications in the way they're changing their databases, etc.

Security has got to be a consideration for DevOps. Question whether or not the way you're changing and managing data in the database is secure. You need to be intentional about integrating security and data into your DevOps process. Do the appropriate level of automation and rigor to make them a part of your process. If you don't pay attention to it, then neither security or data will be part of your DevOps methodology, and you won't get the efficiency, speed, or quality you really want.

What are the industries will immediate returns from Data DevOps, if they do it well?

It's probably easier to describe organizations or industries that won't because I think that's how broad the implications will be. Industries that can wait on this are very slow-moving.

I think any industry that is trying to increase innovation for competitive, cost-cutting reasons, or efficiency reasons will benefit. If you're competing against Amazon, which most people are, then you're going to see benefits, certainly, in financial services and retail. We have a surprising number of logistics customers because they're trying to increase the speed at which they're innovating.

Cycle times require progressive companies to pay attention to their data and database. It's an initiative that has a very broad spectrum of benefits. It's really dependent on the use case. I encourage organizations to not worry so much about their industry but to look at their application portfolios and identify the applications where speed and quality matter the most and start there. It's less industry-specific and more like an application portfolio. Look at your higher-value applications, and go from there.

What are a couple of use cases you'd like to highlight where you've been able to help clients implement Data DevOps to make a positive impact on their business?

Colonial Life Insurance was trying to integrate, increase, and double the number of releases they had on a particular application on an annual basis. The productivity of their sales organization and the ability to release capabilities twice as often is very meaningful. It's not just about cutting costs; it's about increasing revenue.

To be able to do that, their database changes actually added some 1,000 changes. They weren't able to do this manually, so they looked at Data DevOps. We gave them increased speed, quality, security, and a great degree of productivity. We also enabled them to double the number of releases they had on a manual basis.

How does Data DevOps help developers?

Initially and intuitively, you think this is really a database problem, so you work with DBAs, etc. One of our customers told me a couple of years ago that the ROI and benefits of Data DevOps are for developer productivity. He told me their developers spent too much time shepherding releases, worrying about data, worrying about the database and release configurations. He said, "If I can give them that time back, even if it's only a 5% productivity increase because of how much time they're spending on this, it's a huge win for them."

The second part is getting faster feedback on releases because we've automated the database. Getting more rapid feedback allows the developer to develop in a more agile way with greater quality and productivity.

So it's two things. It's taking away some manual work developers previously had to do with going back and forth with database teams to change the databases. And then, because of the automation, we're providing, it gives feedback to the developer much more quickly. The biggest ROI in Data DevOps is for the developer, not the database teams.

What do developers need to do to facilitate the adoption of a data DevOps methodology? What can they do to help?

Push the envelope. Developers are in the best place to recognize the opportunity and need for innovation. So, I think helping their operations counterparts look at a different way to do this is important. Second is the recognition that if you're going to do Data DevOps well, developers are going to have to do some things differently from their normal day-in and day-out basis.

Right now, they may be opening tickets to get data changes, receive data, or have test data populated. They need to embrace automation. They may need to begin putting database changes right into the source code.

People want the benefits of DevOps and Data DevOps. But they don't want to change anything. The reality is that they're going to have to change some of the ways they do their jobs and their workflows to get the benefits. The payoff is huge. I recommend embracing the change that's going to come and then helping drive those changes organizationally.