Creating a Highly Available K3s Cluster

K3s is a lightweight Kubernetes distribution, which is easy to install. It is a fully compliant Kubernetes distribution with an embedded SQLite database as the default datastore and supports external datastore such as PostgreSQL, MySQL, and etcd. K3s includes a local storage provider, a service load balancer, a Helm controller, and the Traefik ingress controller. It also automates and manages complex cluster operations such as distributing certificates. K3s are easier to use, and more lightweight, with a binary size of less than 100 MB. In other words, you can run a highly available, certified Kubernetes distribution designed for production workloads on nanodes as well.

However, for a highly available K3s cluster with two master nodes, you can use an external datastore and an external load balancer for balancing the TCP traffic on 80/http, 443/https, 3306/mysql, and 6443/Kubernetes API.

K3s Highly Available Architecture

Infrastructure Requirements

• Two Linux nodes, typically virtual machines. You can deploy a K3s cluster on just about any flavor of Linux. However, K3s are officially tested on Ubuntu 16.04 and Ubuntu 18.04. Nanode costs just 5$ a month. In this tutorial these nodes are referred as master 1 and master 2.
• An external database to store the cluster data. MySQL is recommended. For more information, see installing MySQL. In this tutorial this external database is referred as mysql.example.com.
• A load balancer or NodeBalancer to direct traffic to the two nodes. Add the configurations for 80/http, 443/https, 3306/mysql, and 6443/Kubernetes API. For more information see, adding a TCP NodeBalancer. A NodeBalancer costs 10$ a month.
• A DNS record to map a URL to the NodeBalancer. This is the application server URL, and downstream Kubernetes clusters need to reach it. For more information, see adding a DNS record. In this tutorial the URL is referred as k3s.example.com.

Before You Begin

• Ensure that all the nodes are in the same region.
• Ensure that mysql.example.com and k3s.example.com resolve to the IP address of the NodeBalancer.
• Ensure that you can connect to the external database mysql.example.com from  master 1 and master 2.

Creating a Kubernetes Cluster 

1. Connect to master 1, and run the following commands to start the K3s server and connect it to the external datastore:

$export K3S_DATASTORE_ENDPOINT="mysql://user:password@tcp(mysql.example.com:3306/rancher

$curl -sfL https://get.k3s.io |
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 -t agent-secret --tls-san
k3s.example.com --node-taint k3s-controlplane=true:NoExecute" sh -s -
server

Where:

• user: the user that you created on mysql.example.com
• password: the password that you set for the user
• rancher: the database that you created on mysql.example.com
• k3s.example.com: the URL to access the application server
• agent-secret: the secret token to use for authentication

2. Connect to master 2, and run the following commands to start the K3s server and connect it to the external datastore:  

$export K3S_DATASTORE_ENDPOINT="mysql://user:password@tcp(mysql.example.com:3306/rancher

$curl -sfL https://get.k3s.io |
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 -t agent-secret --tls-san
k3s.example.com --node-taint k3s-controlplane=true:NoExecute" sh -s -
server


3. To confirm that K3s has been set up successfully, run the following command on either of the K3s server nodes:

$sudo k3s kubectl get nodes 

          The output is similar to:

xxxxxxxxxx

NAME      STATUS   ROLES    AGE    VERSION

master    Ready    master   168m   v1.18.2+k3s1

master2   Ready    master   61m    v1.18.2+k3s1

       Result: You have successfully set up a highly available K3s Kubernetes cluster.

Note: Because K3s server nodes are schedulable by default, the minimum number of nodes for an HA K3s server cluster is two server nodes and zero agent nodes. However, you can always add agent nodes designated to run your apps and services to your cluster.