The Why and When of GitOps

For the last couple of years, the Provectus team has worked with many different customers, from small startups to huge enterprises, to build a wide range of solutions using Kubernetes and the cloud. With all this experience, I’ve realized that the GitOps approach for configuration management is, probably, the best for most of the projects on there.

Existing Problems

All existing approaches to infrastructure configuration management have issues. Some of them just can't be solved by "Infrastructure as Code." For example:

• Implicit infrastructure, its state, and configuration. Sometimes, you have to conduct a full-scale investigation to find out where and how your application is running. 

• Lack of change in history. It can be even harder to figure out how some applications were developed in the first place and who manages them now.

• Security holes in environments. Created for the sake of easier admin access to Continuous Delivery tools, these holes ensure velocity and convenience of development but endanger both applications and their users.

• The dependency of operations success on the previous state. A truly unpredictable variable, which, nonetheless, implicates the success of the entire project.

• Difficulties of disaster recovery. With manual/half-manual processes, disaster recovery requires a strategy in place, which isn’t always the case.

• Eventual consistency of the configuration. Even with the use of the centralized configuration, the actual configuration and a declared one can drift with manual operations.

Fortunately, the GitOps approach can help us resolve all of these issues, provided its principles are properly implemented.

What Is GitOps?

"GitOps" is a term originally coined by Weaveworks in 2017 to define the new method of using developer tooling to drive operations. It’s often associated with Kubernetes configuration, because Kubernetes is used everywhere now, and GitOps is best to manage k8s configuration.

GitOps Described in Bullets:

• git is a single source of truth

• all operations are performed using Pull Requests

• declarative configuration only is used

• push/pull-based automation by default

• it’s the best configuration option for Kubernetes

• but it’s also OK’ish for Terraform and Ansible

This approach is not limited to Kubernetes, or Ansible and Terraform, though. GitOps dictates high-level principles that, if you’d like to, you could use to build a system with bash scripts. 

The Four Main Principles:

1. The entire system has to be defined declaratively

2. Version the whole system definition in Git

3. Automatically apply approved changes to the infrastructure

4. Special automation agents ensure system convergence

“How do we get there?” you might ask. It is an evolutionary process. Let me start with a few examples to illustrate this evolution.

Evolution Examples

Take your granny. Ok? And a shovel. She wants to plant some flowers in the backyard. She digs a bit and plants one flower bed. She’s tired, but it was worth it — the flowers are beautiful. Then, she wants to expand her flower beds and plant ten times more of different kinds of flowers. It's going to be hard to do on her own. Here is where the "Flowers as Code" approach pops up. 

The granny writes instructions on what to do precisely to make the flower bed as the first one. The “army” of grandchildren armed with instructions starts to work on new flower beds. Granny’s now supervising how her instructions are carried out. It still requires some oversight and administrative overhead, but she can scale well — she had guidelines, and if she needs more flower beds, she can “add” more grandchildren.

Now, let’s imagine that hen the flower beds are planted, one of granny's neighbors accidentally ruins one of them. The granny needs to fix the mess, but she also wants to free up some time to focus on flower beds’ design rather than on supervising her grandchildren. So, she buys a fleet of GitOps drones. 

The drones have predefined instructions and provide a user-friendly interface. The granny feeds a flower bed design to them, and they implement it. Now, she can design and enjoy her flower beds at any scale. The drones will take care of the implementation.  

The neighbor, again, accidentally ruins one of the flower beds. But drones immediately fix it according to the desired state, so the granny doesn’t notice the accident.

Putting it into the technical language:

Before 2006, if you needed a LAMP stack up and running, you had to SSH to the server, install or compile packages, make all configs, fix access rights, and start the services. When you needed the second server, you SSH’d again and had to do all the same from scratch, with new versions of packages. If you needed 50 servers, the amount of work would increase by 50x. That's when everybody understood that something needs to change, and the Infrastructure as Code approach became popular. 

The essence of it is that you write repeatable scripts and playbooks, so you can reuse them and, thus, significantly reduce the amount of work you need to do to scale. But you're still responsible for the configuration of servers and focus your energy on how to faster get to the desired state. If something happens on the server after its setup, the server will remain broken until somebody fixes it.

With GitOps, we shift from servers to the desired state, as well as rely on automation to achieve this state. We work with the infrastructure definition in Git, instead of scripts.

The New Paradigm

In the GitOps world, when you need to deploy a new application, you neither bootstrap a new server nor run configuration playbooks. No! What you do is Pull Requests to your infrastructure state repository. After it’s merged, the GitOps automation will take care of all the required configurations, and it’ll make sure that the application stays up and running if something goes wrong.

Bear in mind that we don't care what exactly automation will do. We’re on an entirely new level of abstraction now.

GitOps is a new paradigm, a further step in the evolution of system administration. We shift the focus of daily operations from the infrastructure itself to its representation in the Git repository.

Kubernetes perfectly fits into this new paradigm. Because it’s declarative, and with only one additional step — installing GitOps agent — you can implement the GitOps approach for Kubernetes. GitOps for Kubernetes is best to start experimenting and to get used to the method’s convenience and safety. Next, we’ll be moving all your IaC solutions to GitOps.

GitOps Challenges 

Of course, GitOps is not a silver bullet. It isn't the panacea for all infrastructure problems. It has its challenges. 

The very first challenge you face when deciding to adopt the GitOps approach is the number of decisions to be made. Because GitOps introduces the focus shift and dictates very high-level principles, it makes you decide how you should manage manifests, split repositories, and implement environment provisioning and updates. Automation, in general, is something you have figure out yourself. For Kubernetes, you can find a whole bunch of GitOps agents, such as ArgoCD or Weave Flux — two most mature solutions.

The greater challenge of the GitOps approach is cultural change, however. GitOps reminds me of DevOps since it pushes us to change our habits, mindset, and daily work routines. This focus shift is hard to carry out at first as all developers and system admins are used to working with servers directly. And now we're telling them that all they can do is Pull Request. You can't SSH to the server and apply a hotfix to the application or its configuration. Or, let me put it this way: You can do that, but automation will revert it, all the same, forcing you to follow the process.

This mindset started to shift when everything started to get containerized with Kubernetes. As the application in the container is immutable, you can't just patch it. You have to make PR and build a new image. But that concerned only developers, system administrators could still interact with the system directly and make configuration changes. Now, with the GitOps approach, if you want to change anything — make a Pull Request even if you're the system administrator.

This cultural change is fundamental for the implementation of GitOps. It requires lots of effort from the team and even more effort from leads and managers. Note: This simplified introduction won’t be enough to adopt the GitOps culture top-down.

Rewards for the Effort

So, what are we going to get for all the effort? 

Transparency. With GitOps, you get a self-documenting infrastructure and self-described deployments. You always know where you have deployed something. You have a history of changes with their authors for free. The state of infrastructure is not a sacred knowledge anymore. On top of that, you can create self-service platforms much easier.

Easy rollbacks and disaster recovery. If you have a complete representation of your infrastructure, you can recreate the whole system from scratch in a matter of minutes. And, if your latest update hasn’t gone well, the only thing you need to do is "git revert." 

Improved security. GitOps improves security in many different ways. Change flow and approvals enforcement on the level of Git repo. You can split and divide access in whatever way you want to by different repo structuring. Runtime environment safety — you don't have to pass Kubernetes admin access to anything outside of Kubernetes, because GitOps agent manages the cluster from the inside. The bright illustration of the security advantages of the GitOps approach is that Platform One by DoD utilizes the GitOps approach with the ArgoCD agent. 

Accuracy of operations. We always get what is described in the Git. We don't have to rely on the third party with the script anymore. 

Faster onboarding process. With the entire infrastructure and application configuration in Git, it's much easier to onboard new team members. With proper commit messages, new hires can get familiar with daily operations and processes much faster and easier.

And these aren’t all possible advantages that GitOps can bring to the table. Still,  the list is quite impressive, isn't it?

Conclusion

I hope it's now clear why and when you should be using GitOps. Let's summarize.

When to start implementing the GitOps approach? It depends. Maybe you had to start doing it yesterday; e.g. if you already have a Kubernetes cluster and don't have a clear picture of what and where is deployed.

How to do it? Easy — it's just two steps:

1. Create an infrastructure representation in Git

2. Start to automatically sync infrastructure with its representation 

If you don't know how to better structurize git repo for the Kubernetes cluster, you can check my article on this.

Why use the GitOps approach? Well, to please your granny, of course!