Scaling Continuous Deployment Adoption

Cloud Platforms and cloud-native development have become very prevalent. However, the benefits of cloud-native platforms are realized only when DevOps automation is implemented along with it. The two key processes in DevOps are Continuous Integration (CI) and Continuous Deployment (CD). Most organizations have successfully automated CI over the last decade.  

Continuous deployment helps in delivering software faster with high quality and low risk and improves productivity and efficiency.  However, the automation for CD is a lot less, as per many surveys, due to underlying complexity. Deep knowledge of cloud-native platforms,  architectures, operations, release management, verification, and observability is required for the successful adoption of CD. Multitudes of tools across disciplines, such as CI, CD, security, testing, and observability, have to be orchestrated to make such automation happen. This paper explores the key aspects organizations should consider while adopting continuous deployment and offers guidance to improve the adoption maturity.

Continuous Integration vs Continuous Delivery vs Continuous Deployment

Continuous integration is the process of automating source code management, building and running code quality checks, automated units, and integration tests. Continuous delivery and continuous deployment are extensions of CI. They are often interchangeably used while referring to CD in DevOps. The generally accepted definition of continuous delivery is the process of getting the application packaged and getting it ready for deployment to any environment and usually requiring manual approval for deployment. Whereas continuous deployment is the process of not only getting the application ready for deployment but having it deployed in a fully automated manner without manual workflows. Continuous delivery can be considered as a subset of continuous deployment. 

In this article, we will use the acronym CD to refer to continuous deployment, and most of the points discussed are relevant to continuous delivery as well. Most organizations have successfully automated the CI process and extended the same CI tools to accomplish some aspects of CD. But they are the suboptimal solution as the concerns of CD are much wider beyond CI, as covered in the next section.

Continuous Deployment Capabilities

CD constitutes many capabilities, as summarized in the reference architecture below.  The reference architecture also shows the relations between capabilities across the lifecycle, including CI. 

The key capabilities of continuous deployment are:

• Configurable Pipeline Orchestration: To provide the flexibility to configure different workflows and steps as required by the organization.
• Automated Infrastructure Provisioning: For deploying and configuring the cloud-native platform and cluster services like the Ingress, Service Mesh, Observability tool, etc. 
• Multi-Cloud Deployment: To abstract the cloud-specific differences using cloud provider-specific implementations underneath.
• Multi-Environment Deployment: Support for the ability to promote the deployment to different environments based on the gating check and handle the associated environmental variables.
• Continuous Verification: Of the quality and health of the artifact deployed over a brief period of time and helping in taking the key decision whether to continue or roll back the deployment.
• Multiple Deployment Strategies: There are many deployment strategies for applications requiring high availability and quality of services, such as Blue Green and Canary Deployment Strategies. 
• Security Vulnerability Testing: At various stages of the DevOps cycle, including static assessment during build time and runtime assessment after deployment.
• Automated Policy Enforcement: For enforcing policies such as network policies, resource quotas, container security policies, etc., in an automated manner using Policy as Code.
• Visibility and Insight Into the Metrics and KPIs: To improve the quality of the Software delivery and operational efficiency.
• Auditability and Traceability: To understand who, what and when a particular deployment to production was done and track feature level status across environments.
• Change Management Integration: With service ticketing systems such as Jira and ServiceNow for approval or tracking changes and deployment events. 

Capabilities for Continuous Deployment Adoption

An organization wanting to adopt continuous deployment or increase maturity, if already in the journey, will have to do invest and build capabilities in the following capabilities mentioned below. It need not be an all-or-nothing approach but rather an incremental approach. Organizations can set the target maturity levels for each of the capabilities, assess where they are currently, and progress incrementally. 

Organizational Readiness and DevOps Culture

Organizations aspiring to adopt continuous deployment need to adopt agile methodology in the true sense for the end-to-end software lifecycle. Fully autonomous cross-functional development teams need to be empowered to not only design, build, and test but also deploys and manage the software in production. A continuous improvement mindset is required, and processes have to be created for continuous improvement beyond focusing only on automation.

Quantifying the software delivery performance metrics such as deployment frequency, lead time for changes, mean time to recovery, and change failure rate (DORA metrics) will help in assessing the current state to set the target for CD adoption. 

Mature Continuous Integration Practices

Continuous delivery and deployment are extensions of continuous integration. Hence strong continuous integration practices have to be in place. The practice of frequently committing the code in small increments and automatically triggering the build along with unit and integration tests is important for having the components integrated on an ongoing basis with stability. Beyond these regular build, unit, and integration automation, several code quality and security vulnerabilities, such as Software Bill of material scanning, Open source software scanning, and container image scanning, can be integrated with the CI pipeline itself, which will improve the quality and robustness of the software being built.

Cloud Native and Platform Expertise

In order to achieve continuous deployment, the application architecture and the underlying deployment platform should support it. A monolithic application typically does not support faster releases of only the components that are changed. Microservices architecture is more conducive to continuous deployment as specific components undergoing changes can be deployed separately without impacting the other components. Similarly, a cloud-native platform, whether in the public cloud or in the private cloud, provides the agility and dynamism that are required for continuous deployment. The cloud-native ecosystem is very vast. Many of the associated technologies, such as databases, messaging platforms, orchestration platforms, API gateways, service meshes, storage, network, etc., all have to cloud-native manifestations that can be leveraged for full exploitation. Cloud Native Computing Foundation’s Cloud Native landscape provides a view of the landscape having 700+ tech stacks grouped into several categories. An organization planning to adopt continuous deployment should build expertise in the cloud native ecosystem that is appropriate for them, as it has some implications in continuous deployment. 

Adopting Everything as Code Wherever Possible

An ‘Everything as code’ approach treats all aspects of the system as code and applies the application development approach to it. It brings in automation over the tasks that are usually done manually and thereby avoids manual errors. It provides self-documentation of the tasks that are codified, allows version control, and makes them repeatable any number of times in a reliable manner. The codified actions can be automatically performed by integrating with the continuous deployment pipeline.

Some of the most commonly codified items are below:

• Infrastructure as Code: Declarative, immutable infrastructures are more widely being adopted using tools such as Terraform, Ansible, Pulumi, etc.
• Security as Code: Codifies security policies such as access decisions, detecting security vulnerabilities, bugs, etc.
• Policy and Compliance as Code: Where any kind of policies, including and beyond security policies such as standards compliance, and operational policies, can be codified, version controlled, and enforced. Open Policy Agent (OPA) is becoming a standardized way to define policy using its high-level declarative language.
• Configuration as Code: Cloud-native platform, services, and application configurations can be codified and applied in an automated manner.
• Database Changes: Can be codified using appropriate DB scripts and executed with tools such as Liquibase for smooth propagation of DB changes.

GitOps is a prescriptive style of infrastructure as code for the cloud-native applications that are becoming very popular. GitOps mainly focuses on ensuring that the desired state of the system in the Git and the target deployment state are always in sync. While GitOps originated as the evolution of Infrastructure as Code and infra provisioning automation, GitOps principles have been extended to application configuration management and deployment to leverage the benefits. GitOps, by definition, covers a subset of the CD. Organizations adopting GitOps will have to supplement it with other building blocks for full-fledged CD.

Continuous Verification and Observability

The CD process does not end immediately after the artifact is deployed in production. Verification of the health of the artifact deployed for a period of time and helping in taking the key decision of whether to continue or roll back the deployment is a key aspect of robust CD automation. The following are the different types and levels of verifications. 

• Metrics Monitoring:  To observe the health of the Platform for metrics like CPU, memory utilization, and storage utilization. Tools like Prometheus are prominent for observing the Cloud Native platform.
• Non-Functional Monitoring: To observe the Performance, including latency and throughput. Resiliency monitoring using Chaos testing and security run-time testing using DAST, Fuzzy techniques. 
• Custom Business KPI Monitoring: To observe business functionality and context-specific parameters such as the number of specific business transactions that are successful/failed over a period of time etc.

Determining the parameters to observe is key. While the parameters for metrics and non-functional monitoring can be common across industries, business KPI monitoring is unique to the specific business domain and organization. Thresholds will have to be baselined for the parameters, and action to be taken on the deviation has to be planned. This capability is key for automated decisioning and hence canary and progressive delivery deployment strategies. The use of AI/ML in automated continuous verification decisioning is the latest trend.

Right Adoption of Tools

The continuous deployment tools landscape is very vast, and a holistic solution may require many of the build blocks as depicted in the reference architecture. Each of the building blocks can be addressed by integrating several tools, each covering several or one area. The Continuous Delivery Foundation has a well-curated list of continuous delivery landscapes. There are many continuous deployment tools with differentiating capabilities and strengths in different areas. The tools have different origins and, at a high level, can be clustered into the following:

• Tools with strong capabilities in pipeline and Events: such as Tekton, JenkinsX, Keptn, etc.
• GitOps-based tools with strong capabilities in synchronizing Infrastructure as Code and Configuration from Git to Kubernetes-based target platforms: such as Flux, Argo CD, etc. 
• Cloud Platform CD tools: that are optimized for specific public cloud/cloud platform providers: such as AWS CodePipeline and CodeDeploy, Google Cloud Deploy, Azure Pipelines, Openshift pipeline, etc.
• Tools that have capabilities across multiple areas, including pipelines, multiple deployment strategies, continuous verification, etc, such as Spinnaker, Harness, etc.
• Commercial tools build on top of Open source solutions with some differentiated offerings such as OpsMx, Armory on top of Spinnaker, Codefresh on top of Argo, etc.
• DevOps solution providers such as CloudBees, and Atlassian have CD offering closely integrated with their other DevOps offerings.

 Choosing the appropriate tool can be quite challenging. Similar to any other product selection, identifying the features for CD that is of priority and mapping to the products having strength in those areas will be the optimal approach to tool selection.

Progressive Delivery

Progressive Delivery is the next evolution of the deployment strategy and continuous deployment with an emphasis on feature-based release to a subset of target users. The business KPIs and hence the derived feedback from the target environment and the end user is continuously observed, and rollout is progressively increased for more users or decreased for further analysis and refinement. progressive delivery increases the speed and quality of the delivery further and reduces the risk.  

Feature flags are an essential part of progressive delivery, but that alone is not sufficient. The continuous deployment tools and feature flags management should work closely hand-in-hand. The CD pipeline should be able to accept flags as input and should be able to show the feature flags that are configured in the current release. In a cloud-native solution, the traffic routing required for progressive delivery is addressed by Service Mesh’s such as Istio, LinkerD, etc. Observability discussed earlier, is another key building block for Progressive Delivery. 

Conclusion

DevOps automation is crucial for realizing the benefits of cloud-native platforms. While the adoption of continuous integration is mature in many organizations, the same cannot be said for continuous deployment. Automation in the area of continuous verification, security and compliance, and progressive delivery are important to reaching CD maturity. This article explored the CD capabilities and the capabilities that organizations need to build/enhance in order to adopt continuous deployment and reap the benefit. Organizations can set the target maturity levels for each of the capabilities and progress incrementally.