Using a Custom CockroachDB Image With Docker and Kubernetes
Learn how to build a custom image based on CockroachDB which then can be used in container orchestration of choice.
Join the DZone community and get the full member experience.
Join For FreeMotivation
Cockroach Labs ships new images when a new maintenance release is available, typically on monthly basis. CockroachDB does not rely on the base OS image for any third-party libraries except for geospatial and kerberos packages. That said, OS images may be vulnerable to security exposures depending on their age. Given CockroachDB is written in Golang, replacing the OS can be a trivial task.
Originally, CockroachDB image was shipped with a Debian OS image. CRL then switched to UBI to accommodate a wider scope of use cases including Red Hat OpenShift. UBI images are shipped regularly with CVE patches and given the nature of security vulnerabilities, it is common that the latest and greatest images may or may not have CVEs. Given the preamble, we're going to cover how to replace the base image for CockroachDB and use it in Kubernetes.
High-Level Steps
- Build an image
- Push this image to a container registry
- Verify with Docker
- Clean up
- Verify with Kubernetes
- Clean up
Step by Step Instructions
Build an Image
The current image source code for CockroachDB is located here. I'm going to replace the registry.access.redhat.com/ubi8/ubi-minimal
image with ubuntu:22.04
image from Canonical. The good news is that there are not that many changes I need to make. Given CockroachDB used to be based on Debian, we can use the current and the former images as templates for our own custom image. After several changes, I ended up with the following:
FROM cockroachdb/cockroach:latest-v21.2 AS base
LABEL maintainer="artemervits at gmail dot com"
LABEL version="1.0"
LABEL description="cockroach base image"
LABEL REFRESHED_AT $(date)
FROM ubuntu:22.04
# For deployment, we need the following additionally installed:
# tzdata - for time zone functions; reinstalled to replace the missing
# files in /usr/share/zoneinfo/
# hostname - used in cockroach k8s manifests
# tar - used by kubectl cp
RUN apt-get update && apt-get install -y \
tzdata \
hostname \
tar \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir /usr/local/lib/cockroach /cockroach /licenses /docker-entrypoint-initdb.d
COPY --from=base /cockroach/* /cockroach/
COPY --from=base /licenses/* /licenses/
COPY --from=base /usr/local/lib/cockroach/* /usr/local/lib/cockroach/
# Set working directory so that relative paths
# are resolved appropriately when passed as args.
WORKDIR /cockroach/
# Include the directory in the path to make it easier to invoke
# commands via Docker
ENV PATH=/cockroach:$PATH
EXPOSE 26257 8080
ENTRYPOINT ["/cockroach/cockroach.sh"]
We're using a multi-stage process to generate our custom image.
Once we're satisfied with our result, we can build it.
docker build . -t dbist/cockroach:latest-v21.2
Push this Image to a Container Registry
Once it builds, we can push it to our own container registry
docker push dbist/cockroach:latest-v21.2
Remember to mark the repo public if you're using Docker Hub. I ran into issues fetching the image originally until I realized what was wrong.
Verify With Docker
Let's run through the Docker tutorial to verify our image is functional.
docker network create -d bridge roachnet
docker run -d --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080 -v "${PWD}/cockroach-data/roach1:/cockroach/cockroach-data" dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3
docker run -d --name=roach2 --hostname=roach2 --net=roachnet -v "${PWD}/cockroach-data/roach2:/cockroach/cockroach-data" dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3
docker run -d --name=roach3 --hostname=roach3 --net=roachnet -v "${PWD}/cockroach-data/roach3:/cockroach/cockroach-data" dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3
docker exec -it roach1 ./cockroach init --insecure
At this point, let's connect to any of the live containers and verify the OS in use.
docker exec -it roach1 bash
root@roach1:/cockroach# cat /etc/os-release
PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
At this point, I should be satisfied but we only tested in insecure mode and I want to be absolutely sure I didn't break secure mode. For that, I'm going to use Kubernetes, which will also provide another test scenario.
Let's first clean up after ourselves.
Cleanup
docker stop roach1 roach2 roach3 docker container rm roach1 roach2 roach3 docker network remove roachnet
Verify With Kubernetes
I am going to run through the Kubernetes Operator tutorial to be sure our image works.
minikube start --driver=hyperkit
We will need 16GB of memory and my local Docker Desktop defaults to 8GB which is insufficient for us. Passing --driver=hyperkid
alleviates that problem for me on OSX.
You should see something like:
Creating hyperkit VM (CPUs=8, Memory=16384MB, Disk=20000MB) ...
Apply the Operator CRD
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.5.0/install/crds.yaml
Apply the operator
```bash
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.5.0/install/operator.yaml
Change the namespace used by the Operator
kubectl config set-context --current --namespace=cockroach-operator-system
Check the running pods
kubectl get pods
NAME READY STATUS RESTARTS AGE cockroach-operator-manager-77f4957cdf-sd7gz 1/1 Running 0 46s
Initialize and configure CockroachDB
curl -O https://github.com/cockroachdb/cockroach-operator/blob/master/examples/example.yaml
Edit the file with the custom image we built earlier
# Copyright 2022 The Cockroach Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Generated, do not edit. Please edit this file instead: config/templates/example.yaml.in
#
apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
# this translates to the name of the statefulset that is created
name: cockroachdb
spec:
dataStore:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "60Gi"
volumeMode: Filesystem
resources:
requests:
# This is intentionally low to make it work on local kind clusters.
cpu: 500m
memory: 2Gi
limits:
cpu: 2
memory: 8Gi
tlsEnabled: true
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v21.2.5
image:
name: dbist/cockroach:latest-v21.2
# nodes refers to the number of crdb pods that are created
# via the statefulset
nodes: 3
additionalLabels:
crdb: is-cool
# affinity is a new API field that is behind a feature gate that is
# disabled by default. To enable please see the operator.yaml file.
# The affinity field will accept any podSpec affinity rule.
# affinity:
# podAntiAffinity:
# preferredDuringSchedulingIgnoredDuringExecution:
# - weight: 100
# podAffinityTerm:
# labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/instance
# operator: In
# values:
# - cockroachdb
# topologyKey: kubernetes.io/hostname
# nodeSelectors used to match against
# nodeSelector:
# worker-pool-name: crdb-workers
specifically, replace
image:
name: cockroachdb/cockroach:v21.2.5
with
image:
name: dbist/cockroach:latest-v21.2
This is where the Operator is going to fetch the image from a registry, this is why it is important to make sure it is public.
Apply to the cluster
kubectl apply -f example.yaml
Watch the pods
kubectl get pods --watch
When all pods startup, let's create a secure client we can use to connect to the cluster. You can use the default one from the tutorial or use our custom image to create one. I will do the latter.
curl -O https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/examples/client-secure-operator.yaml
Edit the file to replace the image
# Copyright 2022 The Cockroach Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Generated, do not edit. Please edit this file instead: config/templates/client-secure-operator.yaml.in
#
apiVersion: v1
kind: Pod
metadata:
name: cockroachdb-client-secure
spec:
serviceAccountName: cockroachdb-sa
containers:
- name: cockroachdb-client-secure
image: dbist/cockroach:latest-v21.2
imagePullPolicy: IfNotPresent
volumeMounts:
- name: client-certs
mountPath: /cockroach/cockroach-certs/
command:
- sleep
- "2147483648" # 2^31
terminationGracePeriodSeconds: 0
volumes:
- name: client-certs
projected:
sources:
- secret:
name: cockroachdb-node
items:
- key: ca.crt
path: ca.crt
- secret:
name: cockroachdb-root
items:
- key: tls.crt
path: client.root.crt
- key: tls.key
path: client.root.key
defaultMode: 256
Now we can apply it to the cluster and connect
kubectl apply -f client-secure-operator.yaml
kubectl exec -it cockroachdb-client-secure -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs --host=cockroachdb-public
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.2.5 (x86_64-unknown-linux-gnu, built 2022/02/07 21:01:07, go1.16.6) (same version as client)
# Cluster ID: 372663d8-2e89-4f48-9afd-674e9c5405d5
#
# Enter \? for a brief introduction.
#
root@cockroachdb-public:26257/defaultdb>
At this point, we confirmed we can connect to the database and the cluster is up. Let's shell into a pod and confirm OS image is the one we expect.
kubectl exec -it cockroachdb-1 -- bash
I have no name!@cockroachdb-1:/cockroach$ cat /etc/os-release
PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
This wraps it up for us. Hope you found this tutorial useful. We can clean up after ourselves now.
Cleanup
minikube stop
minikube delete
Published at DZone with permission of Artem Ervits. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments