Source Code Management for GitOps and CI/CD

This is an article from DZone's 2023 DevOps Trend Report.

For more:

Read the Report

GitOps has taken the DevOps world by storm since Weaveworks introduced the concept back in 2017. The idea is simple: use Git as the single source of truth to declaratively store and manage every component for a successful application deployment. This can include infrastructure-as-code (e.g., Terraform, etc.), policy documents (e.g., Open Policy Agent, Kyverno), configuration files, and more. Changes to these components are captured by Git commits and trigger deployments via CI/CD tools to reflect the desired state in Git.

GitOps builds on recent shifts towards immutable infrastructure via declarative configuration and automation. By centrally managing declarative infrastructure components in Git, the system state is effectively tied to a Git commit, producing a versioned, immutable snapshot. This makes deployments more reliable and rollbacks trivial. As an added benefit, Git provides a comprehensive audit trail for changes and puts stronger guardrails to prevent drifts in the system. 

Finally, it promotes a more consistent CI/CD experience, as all operational tasks are now fully captured via Git actions. Once the pipeline is configured, developers can expect a standard Git workflow to promote their changes to different environments. 

Even though the benefits of GitOps are well documented, best practices for implementing GitOps are still being formed. Afterall, the implementation details will depend on the nature of the existing code repositories, size and makeup of the engineering teams, as well as practical needs for imperative changes (e.g., emergency rollback or break glass procedures). 

In this article, we'll look at how to choose the best strategy for embracing GitOps with the aforementioned considerations in mind. 

Source Code Management

The first consideration to adopting GitOps is deciding what artifacts to store in which Git repository. Some tech companies, such as Google and Facebook, are staunch supporters of monorepos, utilizing sophisticated tools to build and deploy code. Others take the opposite approach, using multiple repos to segregate applications or products to manage them separately. 

Fortunately, GitOps is not tied to a particular framework, but unless your organization already has robust tooling to deal with monorepo builds, such as Bazel, the general recommendation is to at least separate application source code from deployment artifacts.

Figure 1: Typical GitOps flow

The benefits to separating the two repositories are multifold: 

1. Deployment cadence for application and infrastructure changes can be more easily separated and controlled. For example, application teams may want every commit to trigger a deployment in lower environments, whereas infrastructure teams may want to batch multiple configuration changes before triggering a deployment (or vice-versa).
2. There may be compliance or regulatory requirements to separate who has access to deploy certain aspects of the application stack as a whole. Some organizations may only allow a Production Engineering or SRE team to trigger production deployments. Having separate repos makes access and audit trails easier to configure.
3. For applications with dependent components that necessitate deployment as a single unit, a separate configuration repo allows multiple application or external dependency repos to push changes independently. Then the CD tool can monitor a single configuration repo and deploy all the components at the same time.

With that said, the exact point of separation for what belongs in the application repo versus the deployment artifacts repo depends on the composition of your team. Small startups may expect application developers to be responsible for application code and other deployment artifacts (e.g., Dockerfile, Helm charts, etc.). In that case, keeping those manifests in the same repo and just keeping Terraform configs in another repo may make sense. 

As for larger organizations with dedicated DevOps/infrastructure teams, these dedicated teams may own Kubernetes components exclusively and maintain those separately from application code. 

Deployment Repo Setup

The logical next question is, how many repos should hold deployment configs? The answer to this question is driven by similar considerations as above but with influence from the infrastructure topology. 

• For small teams with a small number of cloud accounts/projects, it will be easier to have a single repo to host all deployment configs to trigger deployments to a small number of environments. As the infrastructure evolves, non-prod artifacts can be separated from prod repos.
• For mid-sized teams with slightly more sophisticated tooling and complex cloud infrastructure (e.g., multiple projects nested under organizations, hybrid-/multi-cloud), a repo per team may work well. This way different controls can be implemented based on security or compliance needs.
• At the other end of the spectrum, a repo per service and environment provides the most flexibility in terms of controls for large teams with robust tooling.

Other CI/CD Considerations 

The main goal of GitOps is to leverage the power of Git to store the single source of truth in terms of the deployed state. In practice, however, what constitutes the versioned, immutable artifact will be determined by the state of CI/CD tools for each team. For teams with slow pipelines, it may be undesirable to trigger frequent updates. Likewise, for product teams needing to interface with external constituents, versioned releases may need to be coordinated with business partners. In such cases, designing the CI pipeline to update the configuration repo on tagged releases may be beneficial. On the other hand, for agile teams with robust CD tools for canary releases, frequent deployments may be favored to collect real-time user feedback. 

Another critical component to call out with GitOps is secret management. Since secrets can't be checked into Git as plaintext, a separate framework is required to deal with secrets. Some frameworks like Bitnami Sealed Secrets check encrypted data into Git and use a controller/operator on the deployed environment to decrypt the secrets. Others separate secrets entirely and leverage secret stores such as Hashicorp Vault. 

Finally, it's important to call out that even with GitOps, configuration drift may still occur. In fact, for small teams with immature tooling, it may be critical to leave some room for imperative changes. For time-critical outages, manual fixes may be necessary to restore the service first before running through the official pipeline for a long-term fix. Before enforcing stringent policies, test rollback and restore strategies to ensure that GitOps systems do not hinder emergency fixes. 

Conclusion

GitOps is the best thing since configuration as code. Git changed how we collaborate, but declarative configuration is the key to dealing with infrastructure at scale, and sets the stage for the next generation of management tools.
– Kelsey Hightower

GitOps applies the best practices that application developers have learned over the years from interacting with Git to declarative configuration and infrastructure. GitOps produces a versioned, immutable state of the system with an audit trail via a tried-and-true system with which developers are already familiar. In this article, we walked through some considerations for implementing GitOps in your organization based on team size, composition, and state of your CI/CD tooling. For some, a monorepo holding everything in one place may work best, whereas for others, granular controls enforced at multiple repos are a requirement. 

The good news is that GitOps is flexible enough to adapt as your needs change. There is a plethora of GitOps tooling available in the market today. Start with a simple tool to reap the benefits of GitOps as you grow your technology stack to boost developer productivity. 

References:

• "5 GitOps Best Practices" by Alex Collins
• "Guide to GitOps," Weaveworks

This is an article from DZone's 2023 DevOps Trend Report.

For more:

Read the Report