CockroachDB With Kerberos and Docker Compose

Articles covering CockroachDB and Kerberos

I find the topic of Kerberos very interesting and my colleagues commonly refer to me for help with this complex topic. I am by no means an expert at Kerberos, I am however familiar enough with it to be dangerous. That said, I've written multiple articles on the topic which you may find below:

• Part 1: CockroachDB with MIT Kerberos
• Part 2: CockroachDB with Active Directory
• Part 3: CockroachDB with MIT Kerberos and Docker Compose
• Part 4: CockroachDB with MIT Kerberos and custom SPN
• Part 5: Executing CockroachDB table import via GSSAPI
• Part 6: CockroachDB, MIT Kerberos, HAProxy, and Docker Compose
• Part 7: CockroachDB with Django and MIT Kerberos
• Part 8: CockroachDB with SQLAlchemy and MIT Kerberos
• Part 9: CockroachDB with MIT Kerberos using a native client
• Part 10: CockroachDB with MIT Kerberos along with cert user authentication
• Part 11: CockroachDB with GSSAPI deployed via systemd
• Part 12: Selecting proper cipher for CockroachDB with GSSAPI
• Part 13: Overriding KRB5CCNAME for CockroachDB with GSSAPI

As our customers are increasing their footprint and considering production use cases, I'm being asked to walk through the typical steps of enabling Kerberos auth for Cockroach. As this process is pretty heavy-handed, a few of us at Cockroach had sought out a repeatable process in getting an on-demand environment quickly and efficiently. CockroachDB source code is a good starting point for learning the inner workings of CRDB. I knew there were compose recipes available with Kerberos to test through the integration but typically they are written for Go language tests. We decided to introduce our own docker compose with nothing but a Kerberos realm, an instance of CockroachDB, and a Postgres client to connect with. The last part is only necessary for a bit longer as we're actively working on building GSSAPI support into the cockroach CLI.

So in a nutshell, we need three containers, an instance of Cockroach (can be many), an instance of Kerberos realm and psql client.

1. First, we need to set up a secured instance of Cockroach, we will need certs generated to make this happen. There are actually four containers that we need to stand up but only three will be active to the end-user. I will intentionally leave the details out of this as you can review the code for roach-cert container in my repo.

2. Next, we're going to stand up an instance of Kerberos, that's pretty much word for word copy of the kdc container from our CockroachDB repo above with slight modifications to add a hard-coded IP for the node.

3. Cockroach container is next and here all I'm doing in addition to my previous posts on the topic is adding an environment variable KRB5_KTNAME according to our GSSAPI docs, mapping a keytab that will be generated by the kdc container and again a hard-coded IP for the node.

4. Finally, we need an instance of the PostgreSQL client, which we can just grab as a published docker hub image and mount the certs, keytab, and Kerberos config to it.

version: '3.8'

services:

  roach-cert:
    container_name: roach-cert
    hostname: roach-cert
    build: roach-cert
    volumes:
      - certs-cockroach:/certs/cockroach
      - certs-client:/certs/client

  kdc:
    container_name: kdc
    hostname: kdc
    build: ./kdc
    volumes:
      - ./kdc/start.sh:/start.sh
      - keytab:/keytab
    depends_on:
      - roach-cert
    networks:
      roachnet:
        ipv4_address: 172.28.1.3

  cockroach:
    container_name: cockroach
    hostname: cockroach
    image: cockroachdb/cockroach:v20.1.3
    depends_on:
      - kdc
      - roach-cert
    ports:
      - "26257:26257"
      - "8080:8080"
    command: start-single-node --certs-dir=/certs --listen-addr cockroach
    environment:
      - KRB5_KTNAME=/keytab/crdb.keytab
    volumes:
      - certs-cockroach:/certs
      - keytab:/keytab
    networks:
      roachnet:
        ipv4_address: 172.28.1.4

  psql:
    container_name: psql
    hostname: psql
    build: ./psql
    extra_hosts:
      - "cockroach:172.28.1.4"
    depends_on:
      - cockroach
      - kdc
      - roach-cert
    environment:
      - PGHOST=cockroach
      - PGPORT=26257
      - PGSSLMODE=require
      - PGSSLCERT=/certs/node.crt
      - PGSSLKEY=/certs/node.key
    volumes:
      - ./kdc/krb5.conf:/etc/krb5.conf
      - ./psql/start.sh:/start.sh
      - certs-client:/certs
      - keytab:/keytab
    networks:
      roachnet:
        ipv4_address: 172.28.1.5

volumes:
  keytab:
  certs-cockroach:
  certs-client:

networks:
  roachnet:
    ipam:
      driver: default
      config:
        - subnet: 172.28.0.0/16

Now let's take a step back for a minute to discuss the need for hard-coded IPs, we need the /etc/hosts file on the psql container to be populated with the IP and hostname of the cockroach container as the exact spelling of the instance will be used as part of the Service Principal Name for Cockroach, postgres/cockroach@EXAMPLE.COM. For that to work, the name needs to be resolvable and without it, it is really hard to debug Kerberos with a cryptic message about GSS not working, trust me I know. That's why I added a fairly new property to this compose file called extra_hosts, which will auto-populate the hosts' file automatically.

    extra_hosts:
      - "cockroach:172.28.1.4"

At this point, we have all of the prerequisites taken care of and we can glue it together.

5. I created a helper file called up.sh to start composing in the background, create a user called tester, grant it necessary permissions in Cockroach, apply Kerberos authentication config and a license, (Kerberos requires enterprise license).

docker-compose build --no-cache
docker-compose up -d

docker-compose exec cockroach \
 /cockroach/cockroach sql \
 --certs-dir=/certs --host=cockroach \
 --execute="CREATE USER tester;"

docker-compose exec cockroach \
 /cockroach/cockroach sql \
 --certs-dir=/certs --host=cockroach \
 --execute="GRANT ALL ON DATABASE defaultdb TO tester;"

docker-compose exec cockroach \
 /cockroach/cockroach sql \
  --certs-dir=/certs --host=cockroach \
  --execute="SET cluster setting server.host_based_authentication.configuration = 'host all all all gss include_realm=0';"

docker-compose exec cockroach \
 /cockroach/cockroach sql \
 --certs-dir=/certs --host=cockroach \
 --execute="SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';" -e "SET CLUSTER SETTING enterprise.license ='${COCKROACH_DEV_LICENSE}';"

6. Run the up.sh script and everything should startup.

7. We can check the status with docker-compose ps command.

➜  kerberos-part1 docker-compose ps
   Name                 Command               State                Ports
--------------------------------------------------------------------------------------
cockroach    /cockroach/cockroach.sh st ...   Up      0.0.0.0:26257->26257/tcp,
                                                      0.0.0.0:8080->8080/tcp
kdc          /start.sh                        Up
psql         /start.sh                        Up      5432/tcp
roach-cert   /bin/sh -c tail -f /dev/null     Up 

8. Connect to the PSQL container and try to access the database

docker exec -it psql bin/sh

psql "postgresql://cockroach:26257/defaultdb?sslmode=require" -U tester

psql (9.5.22, server 9.5.0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

defaultdb=>

9. The root user can still access the cluster with certs as I already had mentioned in my previous blogs.

psql "postgresql://cockroach:26257?sslcert=/certs%2Fclient.root.crt&sslkey=/certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=/certs%2Fca.crt"

10. Let's step through the details of what has to happen behind the scenes for each container, first up, kdc container.

We have a krb5.conf file that we use to build Kerberos realm. For the sake of brevity, feel free to review the code repo for the actual file. The dockerfile uses a standard Alpine image, adds krb5-server library, we're also copying the krb5.conf file into the container and creating and populating the Kerberos database with a principal tester@EXAMPLE.COM and assigning an SPN postgres/cockroach@EXAMPLE.COM.

RUN kdb5_util create -s -P kpass \
  && kadmin.local -q "addprinc -pw psql tester@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey postgres/cockroach@EXAMPLE.COM"

I also have a start.sh file that serves as an entry point for the kdc container. Here all I'm doing is generating a keytab with the principal and the associated SPN.

11. Next up is the roach-cert container, again, feel free to look at the Dockerfile for the details, all I'm doing is using a public cockroach docker hub image, generating certs and copying them into mounted volumes on the containers in the compose file.

12. Finally, the psql container has a Dockerfile with Postgres image and I'm only adding krb5-user library for Kerberos client tools and start.sh file serving as an entry point. In that file, all I'm doing is kinit as tester and running tail indefinitely so that the container doesn't terminate.

At this point, we have a functional environment.

TLDR:

Connecting to the psql container:

docker exec -it psql bin/sh

Connecting to the kdc container:

docker exec -it kdc bin/sh

Interacting with Kerberos:

# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:  list_principals
K/M@EXAMPLE.COM
customspn/cockroach@EXAMPLE.COM
kadmin/9c6b87e87696@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kiprop/9c6b87e87696@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
postgres/cockroach@EXAMPLE.COM
tester@EXAMPLE.COM

Connecting to the cockroach node:

docker exec -it cockroach bash

root@cockroach:/cockroach# cockroach version
Build Tag:    v20.1.3
Build Time:   2020/06/23 08:44:08
Distribution: CCL
Platform:     linux amd64 (x86_64-unknown-linux-gnu)
Go Version:   go1.13.9
C Compiler:   gcc 6.3.0
Build SHA-1:  7fd454f880f386cdd0eda6b21b12f6532c14f0db
Build Type:   release

Disclaimer

Using sslmode=require is susceptible to Man In The Middle (MITM) attack and it is highly encouraged to use verify-ca or verify-full.

verify-ca is not susceptible to MITM attack by verifying the server is trustworthy with the certificate authority

psql "postgresql://cockroach:26257/defaultdb?sslmode=verify-ca&sslrootcert=/certs/ca.crt" -U tester
psql "host=cockroach port=26257 sslmode=verify-ca user=tester krbsrvname=customspn sslrootcert=/certs/ca.crt"

verify-full will validate the server is trustworthy with CA as well as validate the Common Name attribute of the certificate matches the hostname.

psql "postgresql://cockroach:26257/defaultdb?sslmode=verify-full&sslrootcert=/certs/ca.crt" -U tester
psql "host=cockroach port=26257 sslmode=verify-full user=tester krbsrvname=customspn sslrootcert=/certs/ca.crt"

For more information, please read the following doc.

Hope you enjoyed this overview and I intend to take this further in the next post. If you have suggestions and feedback, please don't hesitate to leave a comment or file issues or even contact me at [artem at cockroachlabs.com].