Mutual TLS With gRPC Between Python and Go Services

This tutorial walks you through the process of connecting services written in Python and Go via the gRPC framework using mutual TLS authentication. I assume that the reader is somewhat familiar with Python/Django and Go development and so omit most of the boring stuff like bootstrapping virtualenv with the Django app or how to "manage.py runserver" it.
The final code can be found here.

Introduction

I have an old system in Python undergoing a significant overhaul. It's a two-component system:

• Webapp is a user-facing web application built with the Django framework. It acts as an API client and connects to several nodes to perform some actions.
• Each node (server) is a simple server written in Python that resides behind Nginx. Some of the nodes live outside a private network, and communication happens over a public network.

After some work put into cleanup, refactoring, and testing, the client pretty much satisfies its requirements. On the other hand, the server is trouble - it's poorly written, has some issues with stability, and performance becomes a serious problem. The obvious solution here is to rewrite the server in Go. Go (Golang) doesn't need an introduction and would help with performance.

The only problem here is a communication barrier between Python and Go.

The existing JSON API used for communication between client and server is old and not documented. Exactly the case when it's easier to rebuild it from scratch instead of trying to revive it. To rewrite this API as REST/JSON is relatively easy, but JSON as an exchange format will not provide interchangeability and types compatibility between Python and Go. Types systems are different in those two languages, and it would be tedious and error-prone to make it work.

A better solution here is to use a cross-platform serialization format like protocol buffers (protobuf). It is built to provide cross-platform compatibility and is well supported in Python and Go, and it's also smaller and faster than JSON. Protobuf can be used with REST API to ensure data interoperability between programming languages. But an even better solution is to use the gRPC framework to replace the old API entirely.

The gRPC is a remote procedure call (RPC) framework that works very well in cross-service communication scenarios. It uses protocol buffers as both Interface Definition Language (IDL) and as a message interchange format. gRPC uses HTTP/2 as transport and supports Transport Layer Security (TLS) protocol, and it can work without TLS - basically, it's what most of the tutorials show us. This way, communication is done via h2c protocol, essentially plain-text HTTP/2 without TLS encryption. However, when communication is performed over a public network, TLS is a requirement. And taking into account modern security threats, TLS should be considered even for private networks connections. Source

Service-to-service communication in the case of this system doesn't require distinguishing clients or granting different permissions to them. Still, it's important to ensure only authorized clients speak to the servers. And it's easy to implement using mutual TLS (mTLS) as an authentication mechanism.

Normally in TLS, the server has a certificate and public/private key pair, while the client does not. The server then sends its certificate to a client for verification. In mTLS, both the server and client have certificates, and the server also verifies the client's certificate. Only after that the server grants access to a client. Source 

Let's create something similar - a simple Python/Django web service that will call Go server via gRPC/mTLS and display results in a browser, and start with the repository's structure.

Code Layout

Using a single repository (monorepo) for such a project removes the need to share API schema. Everyone has personal preferences on how to organize a codebase, just keep in mind that Protobuf Compiler, Protoc has its own ideas on how the code should be organized. Placement of the proto files influences compiled code. And it may require some experimenting with compiler flags to generate working code. Put proto files outside the main folders with code, so reorganizing code won't break proto compilation.

I suggest using a directory structure like this:

tree -L 1 -d .
.
├── certs
├── client
├── proto
└── server

• Certs - our Public key infrastructure for self-signed certificates.
• Client - Python/Django web app and also gRPC client for API. It's basically a result of a django-admin startproject client .  with stripped-down configuration because a database is not required.
• Proto - is where to put protobuf source files for gRPC.
• Server - gRPC server in Go.

Public Key Infrastructure

To start with TLS you need certificates for both, the client and server. To create self-signed certificates I suggest CloudFlare's PKI toolkit, CFSSL.

First, you need to create a Certificate Authority (CA) that will be used to generate TLS certificates for server and client. This CA certificate is also used to verify other party certificates' authenticity while establishing a TLS connection.

CFSSL configured via JSON files and provides commands to generate default configuration templates to start with:

cd certs
cfssl print-defaults config > ca-config.json

Default ca-config.json provides profiles that are sufficient for our needs. Let's generate CA certificate signing request configuration, certificate, and private key:

cat > ca-csr.json <<EOF
{
  "CN": "CA",
  "key": {
      "algo": "ecdsa",
      "size": 256
  },
  "names": [
      {
          "C": "US",
          "ST": "CA",
          "L": "San Francisco"
      }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

Client certificate, public and private key:

cat > client-csr.json <<EOF
{
  "CN": "client",
  "key": {
      "algo": "ecdsa",
      "size": 256
  },
  "names": [
      {
          "C": "US",
          "ST": "CA",
          "L": "San Francisco"
      }
  ]
}
EOF

cfssl gencert \\
  -ca=ca.pem \\
  -ca-key=ca-key.pem \\
  -config=ca-config.json \\
  -profile=client client-csr.json | cfssljson -bare client

The server's IP address must be included in the list of subject alternative names for the API server certificate. This will ensure remote clients can validate the certificate.

cat > server-csr.json <<EOF
{
    "CN": "server",
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "ST": "CA",
            "L": "San Francisco"
        }
    ]
}
EOF

cfssl gencert \\
  -ca=ca.pem \\
  -ca-key=ca-key.pem \\
  -config=ca-config.json \\
  -hostname=127.0.0.1 \\
  -profile=server server-csr.json | cfssljson -bare server

That's all for certificates.

Protobuf and gRPC

Now, that the certificates are ready, the next step is to create a schema definition for the API required by gRPC. It is a simple service named DiceService, to demonstrate how gRPC and mTLS work.

Below is the content of the proto/api.proto file. It defines an RPC endpoint RollDie that accepts RollDieRequest and returns the value of rolled die in the value field of RollDieResponse.

syntax = "proto3";

option go_package = "server/api";

package api;

message RollDieRequest {}

message RollDieResponse {
  int32 value = 1;
}

service DiceService {
  rpc RollDie (RollDieRequest) returns (RollDieResponse) {}
}

The next step is to generate code for each language from the proto definition using protobuf compiler - Protoc. In addition, each language requires its own set of dependencies.

Install required packages and build compile proto file for Python:

pip install grpcio grpcio-tools
python -m grpc_tools.protoc -I proto --proto_path=proto \\
    --python_out=client/api --grpc_python_out=client/api proto/api.proto

Compiled files are located in the client/api directory. For some reason, protoc's compiler for Python uses absolute import in generated code and it should be fixed:

cd client/api && cat api_pb2_grpc.py | \\
    sed -E 's/^(import api_pb2.*)/from client.api \\1/g' > api_pb2_grpc.tmp && \\
    mv -f api_pb2_grpc.tmp api_pb2_grpc.py

Install required modules and build a proto file for Go. protoc-gen-go and protoc-gen-go-grpc by default are installed in GOBIN directory. You can override GOBIN and point it to virtualenv's bin directory - it makes it easier to clean after.

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
protoc -I. --go_out=. --go-grpc_out=. proto/api.proto

Compiled files are located in the server/api directory.

Python Client

To isolate the rest of the codebase from Protobuf/gRPC code, create a simple wrapper: api/client.py. This wrapper expects a CA certificate, clients certificate, and key to establish a TLS connection to the provided address.

import grpc

from . import api_pb2, api_pb2_grpc

class Certs:
  root = None
  cert = None
  key = None

  def __init__(self, root, cert, key):
    self.root = open(root, 'rb').read()
    self.cert = open(cert, 'rb').read()
    self.key = open(key, 'rb').read()

class Client:
  rpc = None

  def __init__(self, addr: str, crt: Certs):
    creds = grpc.ssl_channel_credentials(crt.root, crt.key, crt.cert)
    channel = grpc.secure_channel(addr, creds)
    self.rpc = api_pb2_grpc.DiceServiceStub(channel)

  def roll_die(self) -> int:
    return self.rpc.RollDie(api_pb2.RollDieRequest()).value

And here is how to use this client in the web app's view. Variable value here contains the result of the RPC call.

ICONS = ["?", "⚀", "⚁", "⚂", "⚃", "⚄", "⚅"]

def grpc(request):
  grpc_addr = "127.0.0.1:8443"

  crt = api.Certs('certs/ca.pem', 'certs/client.pem', 'certs/client-key.pem')
  try:
    value = api.Client(grpc_addr, crt).roll_die()
  except Exception as e:
    logger.exception(e)

  return HttpResponse('Value: ' + ICONS[value])

Now, if you try to execute this code by starting the web app and hitting the corresponding view, you will get an error. And it is expected - the server is yet to be created. The interesting part here is the error - it will say something about "failed to connect to all addresses", which is not much. But setting the environment variable GRPC_VERBOSITY=debug makes gRPC output more verbose and helps a lot with troubleshooting. It can be done in the client/settings.py file, for example:

if DEBUG:
    os.environ['GRPC_VERBOSITY'] = 'debug'

Go Server

Implement DiceService logic in the server/api/server.go. It initializes the pseudo-random number generator and returns random values in the range from 1 to 6 on request.

// Number of dots on a die
const Dots = 6

type Server struct {
    UnimplementedDiceServiceServer
    rnd *rand.Rand
}

func NewServer() *Server {
    return &Server{
        rnd: rand.New(rand.NewSource(time.Now().UnixNano())),
    }
}

func (s *Server) RollDie(ctx context.Context, req *RollDieRequest) (*RollDieResponse, error) {
    // rand.Intn returns a value in [0, Dots) interval
    value := s.rnd.Intn(Dots) + 1
    return &RollDieResponse{Value: int32(value)}, nil
}

The service implementation is ready. The next step is to provision a gRPC server with certificates and start it. You can put it here, for example:server/server.go An important moment here for enabling mTLS is to set tls.Config{ClientAuth: tls.RequireAndVerifyClientCert} , it indicates to the server to request and validate the client's certificate.

secureAddress := "127.0.0.1:8443"

serverCert, err := tls.LoadX509KeyPair("certs/server.pem", "certs/server-key.pem")
if err != nil {
    log.Printf("failed to load server cert/key: %s", err)
    os.Exit(1)
}

caCert, err := ioutil.ReadFile("certs/ca.pem")
if err != nil {
    log.Printf("failed to load CA cert: %s", err)
    os.Exit(1)
}

caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
creds := credentials.NewTLS(&tls.Config{
    Certificates: []tls.Certificate{serverCert},
    ClientCAs:    caCertPool,
    ClientAuth:   tls.RequireAndVerifyClientCert,
})

secureSrv := grpc.NewServer(grpc.Creds(creds))
log.Printf("Starting gRPC server, address=%q", secureAddress)
lis, err := net.Listen("tcp", secureAddress)
if err != nil {
    log.Printf("failed to listen: %s", err)
    os.Exit(1)
}

api.RegisterDiceServiceServer(secureSrv, api.NewServer())
if err := secureSrv.Serve(lis); err != nil {
    log.Printf("failed to serve: %s", err)
    os.Exit(1)
}

Now run the server (go run server/server.go), make sure the web app runs and visits its url - you should see the result of the RPC request. gRPC server does not log any information about incoming requests, and it is hard to tell what's going on by looking at servers' output. Fortunately, there is an API to intercept the execution of the RPC request, and you can use it to add logging similar to any HTTP server. It's close to how Django's middleware work. Here is a simple logging interceptor. To use it, you need to pass it to the grpc.NewServer.

func loggingInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    ts := time.Now()

    peer, ok := peer.FromContext(ctx)
    if !ok {
        return nil, status.Errorf(codes.InvalidArgument, "missing peer")
    }
    md, ok := metadata.FromIncomingContext(ctx)
    if !ok {
        return nil, status.Errorf(codes.InvalidArgument, "missing metadata")
    }

    res, err := handler(ctx, req)

    log.Printf("server=%q ip=%q method=%q status=%s duration=%s user-agent=%q",
        md[":authority"][0],
        peer.Addr.String(),
        info.FullMethod,
        status.FromContextError(err).Code(),
        time.Since(ts),
        md["user-agent"][0],
    )
    return res, err
}

...

secureSrv := grpc.NewServer(grpc.Creds(creds), grpc.UnaryInterceptor(loggingInterceptor))

It seems that the goal has been achieved. The client speaks to the server via gRPC, and the connection is secure and mutually authenticated, thanks to mTLS. But keep in mind that the server is pretty basic and requires some work and hardening before it can be used in production in public networks.

Let’s Put the Server Behind Nginx

Another approach is to put Nginx before the server, and I like it more than exposing the Go service to the Internet. Out of the box, you will get all battle-tested features like load balancing and rate-limiting, and it will also reduce the amount of code you need to write and support.

Nginx has native support for the gRPC since version 1.13.10 and can terminate, inspect, and route gRPC method calls. So let's add Nginx before the server and let it handle mTLS, and proxy requests via unencrypted HTTP/2. 

This setup is a bit more complicated, so here is the diagram:

Let's start with another view of the client. It will use a different port number.

def nginx(request):
  nginx_addr = "127.0.0.1:9443"

  crt = api.Certs('certs/ca.pem', 'certs/client.pem', 'certs/client-key.pem')
  try:
    value = api.Client(nginx_addr, crt).roll_die()
  except Exception as e:
    logger.exception(e)

  return HttpResponse('Value: ' + ICONS[value])

Because Nginx will do all the work around TLS there is no need to provision gRPC server with certificates in the servers' code:

insecureAddress := "127.0.0.1:50051"
insecureSrv := grpc.NewServer(grpc.UnaryInterceptor(loggingInterceptor))
log.Printf("Starting gRPC server (h2c), address=%q", insecureAddress)
lis, err := net.Listen("tcp", insecureAddress)
if err != nil {
    log.Printf("failed to listen: %s", err)
    os.Exit(1)
}

api.RegisterDiceServiceServer(insecureSrv, api.NewServer())
if err := insecureSrv.Serve(lis); err != nil {
    log.Printf("failed to serve: %s", err)
    os.Exit(1)
}

Here is our config file for Nginx: nginx.conf This config disables demonization and starts a single process with logging to stdout. That's more convenient for demonstration purposes.

events {
    worker_connections  1024;
}

# Do not use it in production!
daemon off;
master_process off;

http {

    upstream grpcservers {
        server 127.0.0.1:50051;
    }

    server {
        listen 9443 ssl http2;

        error_log /dev/stdout;
        access_log /dev/stdout;

        # Server's tls config
        ssl_certificate     certs/server.pem;
        ssl_certificate_key certs/server-key.pem;

        # mTLS part
        ssl_client_certificate certs/ca.pem;
        ssl_verify_client      on;

        location / {
            grpc_pass grpc://grpcservers;
        }
    }
}

Now, all that is remaining to do is to start Nginx!

nginx -p $(pwd) -c nginx.conf

Ensure that's all services are up and visit url for the view created before - you should see the result of the RPC request. 

In case something doesn't work - check the code for this post on GitHub.