How to Secure a Previously Insecure Cluster in Production
Learn how to secure a previously insecure cluster in production and how to mitigate any pitfalls that come with running an insecure cluster in production.
Join the DZone community and get the full member experience.
Join For FreeCockroach Labs does not recommend running an insecure cluster in production. There are only a few additional steps necessary to secure an instance, so why do it? Convenience, you say. It can hurt you down the line but fret not, this article will demonstrate how to fix this. We are going to follow the standard insecure cluster start-up procedure. Once complete, we're going to flip to the documentation for a secure cluster to turn each node on with security enabled. Here's a handy video of the procedure in action:
Step by step instructions are below:
Secure Cluster Take Over
Begin with an insecure cluster.
cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=l ocalhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background * * WARNING: RUNNING IN INSECURE MODE! * * - Your cluster is open for any client that can access localhost. * - Any user, even root, can log in without providing a password. * - Any user, connecting as root, can read or write any data in your cluster. * - There is no network encryption nor authentication, and thus no confidentiality. * * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html * * * INFO: initial startup completed. * Node will now attempt to join a running cluster, or wait for `cockroach init`. * Client connections will be accepted after this completes successfully. * Check the log file(s) for progress. *
bash-3.2$ cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=l ocalhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background * * WARNING: RUNNING IN INSECURE MODE! * * - Your cluster is open for any client that can access localhost. * - Any user, even root, can log in without providing a password. * - Any user, connecting as root, can read or write any data in your cluster. * - There is no network encryption nor authentication, and thus no confidentiality. * * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html * * * INFO: initial startup completed. * Node will now attempt to join a running cluster, or wait for `cockroach init`. * Client connections will be accepted after this completes successfully. * Check the log file(s) for progress.
bash-3.2$ cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=l ocalhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background * * WARNING: RUNNING IN INSECURE MODE! * * - Your cluster is open for any client that can access localhost. * - Any user, even root, can log in without providing a password. * - Any user, connecting as root, can read or write any data in your cluster. * - There is no network encryption nor authentication, and thus no confidentiality. * * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html * * * INFO: initial startup completed. * Node will now attempt to join a running cluster, or wait for `cockroach init`. * Client connections will be accepted after this completes successfully. * Check the log file(s) for progress.
bash-3.2$ cockroach init --insecure --host=localhost:26257 Cluster successfully initialized bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11 CockroachDB node starting at 2020-04-27 13:39:56.808756 +0000 UTC (took 19.4s) build: CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9) webui: http://localhost:8080 sql: postgresql://root@localhost:26257?sslmode=disable RPC client flags: cockroach <client cmd> --host=localhost:26257 --insecure logs: /Users/artem/Downloads/secure/node1/logs temp dir: /Users/artem/Downloads/secure/node1/cockroach-temp775384998 external I/O path: /Users/artem/Downloads/secure/node1/extern store[0]: path=/Users/artem/Downloads/secure/node1 storage engine: rocksdb status: initialized new cluster clusterID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ cockroach sql --insecure --host=localhost:26257 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client) # Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5 # # Enter \? for a brief introduction. # root@localhost:26257/defaultdb> CREATE DATABASE bank; CREATE DATABASE Time: 4.644ms root@localhost:26257/defaultdb> CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMA L); CREATE TABLE Time: 5.005ms root@localhost:26257/defaultdb> INSERT INTO bank.accounts VALUES (1, 1000.50); INSERT 1 Time: 5.354ms root@localhost:26257/defaultdb> SELECT * FROM bank.accounts; id | balance -----+---------- 1 | 1000.50 (1 row) Time: 1.465ms root@localhost:26257/defaultdb> \q bash-3.2$ # connect from another node bash-3.2$ cockroach sql --insecure --host=localhost:26258 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client) # Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5 # # Enter \? for a brief introduction. # root@localhost:26258/defaultdb> SELECT * FROM bank.accounts; id | balance -----+---------- 1 | 1000.50 (1 row) Time: 7.537ms root@localhost:26258/defaultdb> \q
Secure cluster take over:
bash-3.2$ mkdir certs my-safe-directory bash-3.2$ cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key bash-3.2$ cockroach cert create-node localhost $(hostname) --certs-dir=certs --ca-key=my-safe- directory/ca.key bash-3.2$ cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.ke y
Stop each node one by one and restart in secure mode, because each node has the same hostname, we will reuse the cert:
bash-3.2$ ps aux | grep cokroach artem 73363 1.9 0.7 4843760 220372 s003 S 9:39AM 0:05.50 cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 artem 73368 1.0 0.5 4787908 175784 s003 S 9:39AM 0:04.55 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 artem 73371 0.9 0.5 4787176 170988 s003 S 9:39AM 0:04.20 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 artem 73461 0.0 0.0 4417788 840 s003 S+ 9:42AM 0:00.00 grep cockroach bash-3.2$ kill 73363 # stopping node1 bash-3.2$ cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http -addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background bash-3.2$ clear bash-3.2$ ps aux | grep cockroach artem 73371 4.3 0.5 4795368 174144 s003 S 9:39AM 0:04.86 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 artem 73368 3.6 0.5 4796100 177476 s003 S 9:39AM 0:05.18 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 artem 73470 0.9 0.2 4622236 60644 s003 S 9:42AM 0:00.39 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 artem 73476 0.0 0.0 4399356 800 s003 S+ 9:42AM 0:00.00 grep cockroach kill 73368 # node 2 bash-3.2$ cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http -addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background bash-3.2$ clear bash-3.2$ ps aux | grep cockroach artem 73470 2.1 0.3 4700764 86980 s003 S 9:42AM 0:01.51 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 artem 73486 1.8 0.2 4694508 72112 s003 S 9:43AM 0:00.59 cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 artem 73371 0.7 0.5 4797672 178068 s003 S 9:39AM 0:05.85 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 artem 73492 0.0 0.0 4410620 848 s003 S+ 9:43AM 0:00.00 grep cockroach bash-3.2$ kill 73371 # node 3 bash-3.2$ cockroach start --certs-dir=certs --store=node3 --listen-addr=localhost:26259 --http -addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11 CockroachDB node starting at 2020-04-27 13:43:27.451629 +0000 UTC (took 44.3s) build: CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9) webui: https://localhost:8080 sql: postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt RPC client flags: cockroach <client cmd> --host=localhost:26257 --certs-dir=certs logs: /Users/artem/Downloads/secure/node1/logs temp dir: /Users/artem/Downloads/secure/node1/cockroach-temp837519533 external I/O path: /Users/artem/Downloads/secure/node1/extern store[0]: path=/Users/artem/Downloads/secure/node1 storage engine: rocksdb status: restarted pre-existing node clusterID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ # connect to the cluster with client certs bash-3.2$ cockroach sql --certs-dir=certs --host=localhost:26257 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client) # Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5 # # Enter \? for a brief introduction. # root@localhost:26257/defaultdb> show databases; database_name ----------------- bank defaultdb postgres system (4 rows) Time: 2.002ms root@localhost:26257/defaultdb> use bank; SET Time: 993µs root@localhost:26257/bank> use bank; show databases; SELECT * FROM bank.accounts; �[28G\q�[K �[28GSELECT * FROM bank.accounts; id | balance -----+---------- 1 | 1000.50 (1 row) Time: 5.893ms root@localhost:26257/bank> \q bash-3.2$ # connect to another node bash-3.2$ # cockroach sql --certs-dir=certs --host=localhost:26259 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client) # Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5 # # Enter \? for a brief introduction. # root@localhost:26259/defaultdb> SELECT * FROM bank.accounts; id | balance -----+---------- 1 | 1000.50 (1 row) Time: 5.105ms root@localhost:26259/defaultdb> \q
Once complete, you may also validate the certs by creating a user with admin privileges and navigating to the certificates page in the Admin UI, such as https://<address of node with new certs>:8080/#/reports/certificates/local
and view all of the available certificates.
CREATE USER roach WITH PASSWORD 'cockroach'; GRANT ADMIN TO roach; SHOW ROLES; username | options | member_of ----------------+------------+------------ admin | CREATEROLE | {} roach | | {admin} root | CREATEROLE | {admin} (3 rows)
Published at DZone with permission of Artem Ervits. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments