Kops vs. Kubeadm: What's the Difference?

Late last fall, the Weave Online User’s Group (WOUG) had a session on two of the most popular Kubernetes installers out there today. Chris Love (@chrislovecnm) spoke about what kops is and how to use it. Lucas Kaldstrom (@kubernetesonarm) then spoke about the architecture of kubeadm and why you’d choose one method over the other, including the differences between the two projects. Both of our speakers are very active community members and leaders of their respective Kubernetes SIGs.

What is Kops?

Kops stands for Kubernetes operations. The tagline for the project is that it’s “the easiest way to get a production-grade Kubernetes cluster up and running”. Kops is sometimes referred to as the ‘kubectl’ for spinning up clusters.

Kops lets you create, destroy and upgrade Kubernetes clusters and is supported on AWS (Amazon Web Services, we cover more of this on our Kubernetes on AWS - what you need to know page) with GKE in beta support, and VMware vSphere in alpha.

Features include:

• Highly Available (HA) Kubernetes Masters
• A state-sync model for dry-runs and automatic idempotency
• Can generate Terraform
• Support for custom Kubernetes add-ons
• Command line auto-completion
• YAML Manifest Based API Configuration
• Templating and dry-run modes for creating Manifests
• Out-of-the-box support from eight different CNI Networking providers, including Weave Net
• Support for kube-up upgrades
• Ability to add containers, as hooks, and files to nodes via a cluster manifest

Kops on the Google Cloud Platform

According to Chris, using Kops on the Google Container Platform (GCP) is different from spinning up a cluster in Google Kubernetes Engine (GKE) which is Google’s hosted and managed Kubernetes solution.

Chris recommends examining your particular use case before deciding which install option to use. For example, if you have 30 basic microservices, and you are not doing anything special, Google’s managed service, GKE is a viable choice. But if you are running Cassadra stateful sets that are 32 gig heaps, you may need to make some tweaks. For example, if you want to set horizontal pod autoscaling, you can do that in GCP, but not so easily in GKE. If you need a more dynamic installation that can be highly customizable then kops is the product to use with GCP.

Steps for Running Kops on GCP

These are the basic steps for creating a cluster.

1. You need to either use DNS or use Weave Gossip.
2. Create a bucket in storage for the state store.
3. Export your state store: export KOPS_STATE_STORE
4. These are all command line commands you export the feature flag which is alpha and that allows GCE: Export KOPS_FEATURE_FLAGS=AlphaAllowGCE
5. Create the cluster with your project name and your zone set: kops create cluster --name $CLUSTER --project $PROJECT --zones us-central1c --yes

For step by step installation instructions see: Getting Started on GCP with kops

With only three commands you are done and you’ve started creating a cluster. It takes about five minutes for a cluster to be set up. Kops sets up SSH access for you and it also creates the DNS entries. If you are using AWS, kops supports spinning up clusters with manifests as well.

As of today, kops is still not production ready for Google Cloud and it is in Beta. Check their github project for when it will be officially released.

What is Kubeadm?

Lucas Kaldstrom one of the Kubernetes maintainers for kubeadm talked about some of the internals of Kubeadm and also future plans for its ongoing improvements.

Kubeadm is a toolkit for bootstrapping a best-practises Kubernetes cluster on existing infrastructure. Kubeadm cannot provision your infrastructure which is one of the main differences to kops. Another differentiator is that Kubeadm can be used not only as an installer but also as a building block.

Kubeadm sets up a minimal viable cluster. It is designed to have all the components you need in one place in one cluster regardless of where you are running them.

An advantage of kubeadm is that it can be used anywhere —even Raspberry Pi— to set up a cluster and try it out before committing to something like kops.

What a Cluster Created with Kubeadm Looks Like

As mentioned, kubeadm sets up a minimally viable cluster for your use. In the diagram below starting from the bottom, this is what a cluster with kubeadm looks like:

• Infrastructure – at the bottom of the stack, kubeadm assumes that you already have your servers provisioned somewhere in the cloud. It could also be some Raspberry PI hardware sitting on your desk.
• Machines – with machines setup with some type of UNIX, you can install kubeadm onto each machine and then set the control plane (the master) on one machine.
• Bootstrapping – once the master is set up, you’re ready to do the bootstrapping where you connect the nodes to the control plane.
• Kubernetes API – The Kubernetes API is downloaded and initialized.
• Addons – As a last step, kubeadm intends to support Add-ons in the future. Add-ons contain a large number of community contributed projects that cover things like logging, monitoring and load balancers.

Kubeadm vs. Kops

We briefly touched on the differences between the tools earlier. But more specifically these are the differences between the two tools:

• Kubeadm is in the middle of the stack and it runs on each node, and basically creates and then talks to the Kubernetes API.
• Kops on the other hand is responsible for the entire lifecycle of the cluster, from infrastructure provisioning to upgrading to deleting, and it knows about everything: nodes, masters, load balancers, cloud providers, monitoring, networking, logging etc.

The projects are not direct competitors, but instead complement each other.

Creating a Cluster with Kubeadm

Creating a cluster with kubeadm requires only a few commands:

1. Provision several Linux machines with a UNIX flavour
2. Install kubeadm
3. Make one of your machines the master (or the control plane)
4. Install a pod networking layer like Weave Net
5. Join the other nodes to the master

See Using Kubeadm to Create a Cluster for step by step instructions.

To Wrap Up

Chris Love gave us an overview and an update on kops and demonstrated how to spin up a cluster on GCP and on AWS.

Lucas Kaldstrom, a maintainer of Kubeadm, described how kubeadm works and how it differs from kops and how the two projects are complementary.

See also, Kubernetes Custom Cluster: Building Your Own for more information on Kubernetes installation options.

Watch the video in its entirety here:

For more talks like these, join the Weave Online User Group.