Using OpenSSL With libuv

I want to move my simple blocking socket based code to use libuv to allow more than a single connection per thread. The catch is that I also want to do that with TLS, and that seems to be much more challenging. There are a bunch of GitHub projects that talk about this, but as I know nothing about libuv (and very little about OpenSSL), I decided to write my own TLS echo server with libuv to get a better understanding of how it all plays together.

Sit tight, this might take a while to explain. This is a complex topic and it took me a couple of nights of hacking to get it to work, and then, it also took a lot of thinking about how to simplify this into something that I actually like.

There seems to be great documentation for libuv, which is awesome. I went over the simple echo server sample and it seems relatively straightforward. Making the jump to using TLS is a bit harder. OpenSSL makes it really easy to setup SSL on a socket file descriptor and read/write to it. There is even support for non-blocking operations, but I didn’t want to be forced to write my own  select()/poll() code, so how can I integrate these two libraries?

OpenSSL has the notion of a BIO abstraction, which stands for Basic I/O. Basically, this is a stream abstraction. One of the options that OpenSSL has available is the memory BIO. So, the overall idea is to:

• Set up libuv to accept a connection
• Set up OpenSSL with the server side configuration
• When a new connection comes through, set up a new SSL instance from SSL
• Read data from the socket and pass it to the SSL instance and vice versa
• Enjoy encrypted communication

The devil is in the details, naturally. The most complex part, after getting the initial handshake to work, in my experience, is the fact that you can get re-negotiation at any time, which means that a write request will fail with need more read data. That really complicates the amount of state that you have to manage.

Basically, on every SSL_write when managing your own state, you may need to do SSL_read and then retry to previous write. The simplest scenario that we have here is when SSL_accept() on the connection, which results in the following code to manage this state:

void complete_accept_ssl_flush(int socket, write_req_t* req, int status) {
free(req->buf);
free(req);
if (status < 0) {
abort_connection_on_error(socket, status);
return;
}
async_read(socket, cb);
}

void maybe_flush_ssl_buffer(int socket) {
int rc = BIO_pending(state->write);
if (rc > 0) {
write_req_t * req = calloc(1, sizeof(write_req_t));
req->buf = malloc(rc);
req->len = BIO_read(state->write, buf.base, rc);
async_write(socket &buf, 1, complete_accept_ssl_flush);
return;
}
async_read(socket, complete_accept_ssl_flush);
}

void accept_ssl(int socket, void* buf, ssize_t nread) {
if (nread <= 0) {
abort_connection_on_error(socket, nread);
return;
}
BIO_write(state->read, buf->base, nread);
free(buf);
int rc = SSL_accept(state->ssl);
if (rc == 1) {
on_tls_connection_established(client);
return;
}
rc = SSL_get_error(state->ssl, rc);
if (rc == SSL_ERROR_WANT_READ) {
maybe_flush_ssl_buffer(client);
return;
}
abort_connection_on_error(client, rc);
}

To handle a read, we need to check that after every read,  if the act of reading caused us to need to write (client wants to renegotiate the connection, so OpenSSL needs to send data on the connection, which we need to orchestrate) before we can do the actual read. For writes, we need to remember what we are writing, reading, and writing from the network, and then, we repeat our read. This is awkward to do when using synchronous calls, but the amount of state that we have to keep in async and callback-driven programming is a lot. I got it working, but it was really hard and mostly a big house of cards.

I really didn’t like that approach, and I decided that I should go about it in a very different way. I realized that I had a very conceptual error in how I approach libuv. Unlike standard async programming in C#, for example, libuv is based on the idea of a loop. In other words, unlike in the code above, you aren’t going to set up the next read from the network after each one. That is already done for you. You just call un_read_start() , and you’ll get served the data from the network whenever it is available. You can also inject your own behaviors into the loop, which make things really interesting for ourselves.

Here is the logic: we continuously read from the network and pass the buffer to OpenSSL. We then try to read the decrypted data from  SSL_read(). This can fail because we are waiting for more data, and that is fine. We’ll be called again when there is such data. However, we’ll also add a step at the end of the I/O loop to check if there are any pending buffers that needs to be flushed to the network. For writes, if we fail to do the write because we need to read, we’ll register the write to be executed later and wait for the network to send us the read operation.

Given that C isn’t an OO language, I think that I’ll start explaining what is going on from the structs that hold the system together and then the operations that are invoked on them:

typedef struct {
tls_uv_connection_state_t* (*create_connection)(uv_tcp_t* connection);
int (*connection_established)(tls_uv_connection_state_t* connection);
void (*connection_closed)(tls_uv_connection_state_t* connection, int status);
int (*read)(tls_uv_connection_state_t* connection, void* buf, ssize_t nread);
} connection_handler_t;

int connection_write(tls_uv_connection_state_t* state, void* buf, int size);

typedef struct {
SSL_CTX *ctx;
uv_loop_t* loop;
connection_handler_t protocol;
tls_uv_connection_state_t* pending_writes;
} tls_uv_server_state_t;


typedef struct tls_uv_connection_state {
tls_uv_server_state_t* server;
uv_tcp_t* handle;
SSL *ssl;
BIO *read, *write;
struct {
tls_uv_connection_state_t** prev_holder;
tls_uv_connection_state_t* next;
int in_queue;
size_t pending_writes_count;
uv_buf_t* pending_writes_buffer;
} pending;
} tls_uv_connection_state_t;

The first thing to note here is that we have clear layers in the code. We have the  connection_handler_t in here, which is a bunch of function pointers that allow higher level code to work with a connection abstraction. The first portion of the code defines the interface that I expect callers to use. As you can see, we have a few functions that deal with creating, establishing, and tearing down a connection. We also have the most common operations, reads and writes.

The write method is pretty obvious, I think. You give it a buffer and it takes care of writing it to the other side. Note that this is an asynchronous process, and if there are any errors in the process, you’ll get them in the  connection_closed callback. Reading, on the other hand, is completely out of your hands and will be invoked directly by the lower level code whenever it feels like it. This inversion of control may feel strange for people who are used to invoking I/O directly, but it likely allows you to have better overall performance.

Now that we have the interface, let’s build a TLS echo server with it. Here is how that looks like:

tls_uv_connection_state_t* on_create_connection(uv_tcp_t* connection) {
return calloc(1, sizeof(tls_uv_connection_state_t));
}
int on_connection_established(tls_uv_connection_state_t* connection) {
return connection_write(connection, "OK\r\n", 4);
}

void on_connection_closed(tls_uv_connection_state_t* connection, int status) {
report_connection_failure(status);
}
int on_read(tls_uv_connection_state_t* connection, void* buf, ssize_t nread) {
return connection_write(connection, buf, nread);
}

// Basic SSL setup
const SSL_METHOD* method = TLSv1_2_server_method();
SSL_CTX* ctx = SSL_CTX_new(method);
SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM);

// Basic libuv setup
uv_loop_t* loop = uv_default_loop();

// Define the actual behavior of the server
tls_uv_server_state_t server_state = {
.ctx = ctx,
.loop = loop,
.protocol = {
.create_connection = on_create_connection,
.connection_closed = on_connection_closed,
.read = on_read,
.connection_established = on_connection_established
}
};

You can see that there isn’t really much done here. On connection creation, we simply allocate a space for  tls_uv_connection_state_t. This is a callback because your code might want to allocate more space for whatever stuff you want to do in the per connection structure. When the connection is established (after the SSL negotiation, etc), you get a chance to initiate things from the server side. In the code above, we simply let the client know that the connection has been successful. From that point on, we simply echo back to the client anything that they send us.

The SSL and libuv initialization are the bare bones stuff and not really interesting. The nice bits happen in the end of the snippet, where we define the overall server state and wire together the protocol definition.

That is great, but where the part where stuff actually gets done?

A note about this code. I’m writing this primarily for ease of reading/understanding. I’m ignoring a lot of potential errors that, in production code, I would be obliged to handle. That would significantly complicate the code, but it must be done if you want to use this code for anything but understanding the overall concept.

Let’s finish setting up the libuv machinery before we jump to any other code, shall we? Here is what this looks like:

uv_tcp_t server;
uv_tcp_init(loop, &server);
server.data = &server_state;

struct sockaddr_in addr;
uv_ip4_addr("0.0.0.0", DEFAULT_PORT, &addr);

uv_tcp_bind(&server, (const struct sockaddr*)&addr, 0);
int r = uv_listen((uv_stream_t*)&server, DEFAULT_BACKLOG, on_new_connection);
if (r) {
fprintf(stderr, "Listen error %s\n", uv_strerror(r));
return 1;
}
uv_prepare_t after_io;
after_io.data = &server_state;
uv_prepare_init(loop, &after_io);
uv_prepare_start(&after_io, check_if_need_to_flush_ssl_state);

uv_run(loop, UV_RUN_DEFAULT);

This is fairly straightforward. We are listening to a socket and binding any incoming connection to the  on_new_connection() callback. There is also the  after_io preparation stuff, which we use to handle delayed operations (I’ll talk about this later). For now, I want to focus on accepting new connections and processing them.

void on_new_connection(uv_stream_t *server, int status) {
if (status < 0) {
report_connection_failure(status);
return;
}
tls_uv_server_state_t* server_state = server->data;

uv_tcp_t *client = (uv_tcp_t*)malloc(sizeof(uv_tcp_t));
uv_tcp_init(server_state->loop, client);
status = uv_accept(server, (uv_stream_t*)client);
if (status != 0) {
uv_close((uv_handle_t*)client, NULL);
report_connection_failure(status);
return;
}
tls_uv_connection_state_t* state = server_state->protocol.create_connection(client);
state->ssl = SSL_new(server_state->ctx);
SSL_set_accept_state(state->ssl);
state->server = server_state;
state->handle = client;
state->read = BIO_new(BIO_s_mem());
state->write = BIO_new(BIO_s_mem());

BIO_set_nbio(state->read, 1);
BIO_set_nbio(state->write, 1);
SSL_set_bio(state->ssl, state->read, state->write);

client->data = state;

if (server_state->protocol.connection_established(state) == 0) {
abort_connection_on_error(state);
return;
}

uv_read_start((uv_stream_t*)client, alloc_buffer, handle_read);
}

There is quite a lot that is going on this method, and not all of it is obvious. First, we handle accepting the connection and binding its input to the libuv event loop. Then, we create a connection and set up some of the SSL details.

We create an SSL instance for this connection and create two Basic I/O instances that reside in memory. One for the incoming stream and one for the outgoing stream. We’ll be using them to pass data through the OpenSSL encryption, negotiation, etc. We also mark this as a server instance.

Once that is done, we invoke the  connection_established() callback, and then, we tell the libuv event loop to start pumping data from this socket to the  handle_read() callback. For now, I want to ignore the connection_established() callback. It isn’t important to understand the flow of the code at this point (but, we’ll circle back to it). It is important to understand that, by the time we call to this callback, the connection is ready to use and can receive and send data. Well, not receive, because we don’t provide a way to pull data from the connection; we’ll be pushing that data to the provided callback. This will happen by libuv calling to the  handle_read() method whenever there is data on the socket. Here is how we handle this:

void handle_read(uv_stream_t *client, ssize_t nread, const uv_buf_t *buf) {
tls_uv_connection_state_t* state = client->data;

BIO_write(state->read, buf->base, nread);
while (1)
{
int rc = SSL_read(state->ssl, buf->base, buf->len);
if (rc <= 0) {
rc = SSL_get_error(state->ssl, rc);
if (rc != SSL_ERROR_WANT_READ) {
state->server->protocol.connection_closed(state, rc);
abort_connection_on_error(state);
break;
}
maybe_flush_ssl(state);
// need to read more, we'll let libuv handle this
break;
}
if (state->server->protocol.read(state, buf->base, rc) == 0) {
// protocol asked to close the socket
abort_connection_on_error(state);
break;
}
}

free(buf->base);
}
view raw

When libuv calls us with some data, we write this data into the read buffer for OpenSSL, and then, we call  SSL_read() to get the unencrypted data that was sent to us. There are some issues here. First, the SSL/TLS has framing, and the amount of data that your read from the network isn’t going to be the amount of unencrypted bytes that you get in the end. Another issue is that we need to be careful about re-negotiations, which are generally permitted at any point, but they can cause a read to do a write (and may require a write to read).

You might have noticed that this code contains absolutely no indication of this. Instead, we call  SSL_read() to get the plaintext data from OpenSSL. We continue to do this until we get an error from  SSL_read(). This can be either a real error or an indication that we need to read more from the network. Whenever I get some bytes from OpenSSL, I pass them directly to the  read() callback that was provided to us.

If you examine the code carefully, you’ll see that when we run out of data to read, we try to flush the SSL state of the connection. Let’s look at what that method do:

void maybe_flush_ssl(tls_uv_connection_state_t* state) {
if (state->pending.in_queue)
return;
if (BIO_pending(state->write) == 0 && state->pending.pending_writes_count > 0)
return;
state->pending.next = state->server->pending_writes;
if (state->pending.next != NULL) {
state->pending.next->pending.prev_holder = &state->pending.next;
}
state->pending.prev_holder = &state->server->pending_writes;
state->pending.in_queue = 1;

state->server->pending_writes = state;
}

We check if the connection is already in the queue, and if it isn’t, we check whatever should be added. There are two reasons why a connection should be added to the pending_writes queue. First, we may have data buffered in the write buffer of the SSL connection, which needs to be sent over the network. Or, we may have failed writes that we need to retry after we read more data into the SSL connection.

You might notice that we are doing some pointer hopping in the process of registering the connection in the queue. This is basically using a double-linked list and will be important later. If we are putting stuff into a queue, what is going to be reading from this queue?

Remember that when we set up the libuv stuff, we used the  after_io prepare handle? This is called as the first step in the loop, just before we check if there is any I/O to process. This gives us the chance to deal with the confusing read on write and write on read nature of OpenSSL in a more structure manner. Let’s first look at the code, and then, let's see how this all plays out together.

void complete_write(uv_write_t* r, int status) {
tls_uv_connection_state_t* state = r->data;
free(r->write_buffer.base);
free(r);

if (status < 0) {
state->server->protocol.connection_closed(state, status);
abort_connection_on_error(state);
}
}

void flush_ssl_buffer(tls_uv_connection_state_t* cur) {
int rc = BIO_pending(cur->write);
if (rc > 0) {
uv_buf_t buf = uv_buf_init(malloc(rc), rc);
BIO_read(cur->write, buf.base, rc);
uv_write_t* r = calloc(1, sizeof(uv_write_t));
r->data = cur;
uv_write(r, (uv_stream_t*)cur->handle, &buf, 1, complete_write);
}
}

This is what actually handle writing to the network. We take data from the SSL write buffer and send it to the network. Once the write is done, we free buffers that were held for this operation and check if there was any issue with the write (if so, we abort the connection). This is all being driven by this method, which is called before we check for available I/O.

void check_if_need_to_flush_ssl_state(uv_prepare_t* handle) {
tls_uv_server_state_t* server_state = handle->data;
tls_uv_connection_state_t** head = &server_state->pending_writes;

while (*head != NULL) {
tls_uv_connection_state_t* cur = *head;

flush_ssl_buffer(cur);

if (cur->pending.pending_writes_count == 0) {
remove_connection_from_queue(cur);
continue;
}

// here we have pending writes to deal with, so we'll try stuffing them
// into the SSL buffer
int used = 0;
for (size_t i = 0; i < cur->pending.pending_writes_count; i++)
{
int rc = SSL_write(cur->ssl,
cur->pending.pending_writes_buffer[i].base,
cur->pending.pending_writes_buffer[i].len);
if (rc > 0) {
used++;
continue;
}
rc = SSL_get_error(cur->ssl, rc);
if (rc == SSL_ERROR_WANT_WRITE) {
flush_ssl_buffer(cur);
i--;// retry
continue;
}
if (rc != SSL_ERROR_WANT_READ) {
server_state->protocol.connection_closed(cur, rc);

abort_connection_on_error(cur);
cur->pending.in_queue = 0;
break;
}
// we are waiting for reads from the network
// we can't remove this instance, so we play
// with the pointer and start the scan/remove 
// from this position
head = &cur->pending.next;
break;
}
flush_ssl_buffer(cur);
if (used == cur->pending.pending_writes_count) {
remove_connection_from_queue(cur);
}
else {
cur->pending.pending_writes_count -= used;
memmove(cur->pending.pending_writes_buffer,
cur->pending.pending_writes_buffer + sizeof(uv_buf_t)*used,
sizeof(uv_buf_t) * cur->pending.pending_writes_count);
}
}
}

There is quite a lot that is going on in here. First, we iterate through the pending writes for all the connections we have. For each of the connections, we flush the SSL buffer and then check if we have pending writes to process. If we don’t, we can remove the connection from the queue, our work is done. If we do have any pending writes, we need to handle them.

I do that by using  SSL_write(), which will write them into in memory buffer. I continue doing so until one of the following happens:

• I run out of pending writes.
• I run out of buffer space and need to flush.
• I need to re-negotiate and need to read from the network

In the first case, I’ve successfully pushed the data to the SSL buffer, so I can call  flush_ssl_buffer() and then remove the connection from the queue. In the second case, I’ll flush the SSL write buffer and try again.

However, in the last case, I’m just aborting the writes. I need to do a read, and that will be handled on the next iteration of the libuv loop. There is some bookkeeping there to make sure that if we successfully wrote data into the SSL buffer, we won’t be writing that again, but this is pretty much it. You’ll note that I’m playing games with pointers to pointers there to get clean code on the code that consumes the queue but allows me to skip one of the steps in the linked list without removing it from the list.

This is pretty much it, I have to say. We now have a system that both writes and reads work in conjunction to get the proper SSL behavior, even when we have renegotiation going on.

One thing you’ll not find in this code is a call to  SSL_accept() or, indeed, any behavior related to explicitly managing the SSL state. I’m letting OpenSSL handle all of that are rely on the fact that I  SSL_write() and  SSL_read() will handle renegotiations on their own for me.

Let’s do a simple walk through of what is going on with the connection of the TLS echo server.

On connection established (and before we read anything from the network), we call to  connection_write():

int connection_write(tls_uv_connection_state_t* state, void* buf, int size) {
int rc = SSL_write(state->ssl, buf, size);
if (rc > 0)
{
maybe_flush_ssl(state);
return 1;
}

rc = SSL_get_error(state->ssl, rc);
if (rc == SSL_ERROR_WANT_WRITE) {
flush_ssl_buffer(state);
rc = SSL_write(state->ssl, buf, size);
if (rc > 0)
return 1;
}

if (rc != SSL_ERROR_WANT_READ) {
state->server->protocol.connection_closed(state, rc);
abort_connection_on_error(state);
return 0;
}

// we need to re negotiate with the client, so we can't accept the write yet
// we'll copy it to the side for now and retry after the next read
uv_buf_t copy = uv_buf_init(malloc(size), size);
memcpy(copy.base, buf, size);
state->pending.pending_writes_count++;
state->pending.pending_writes_buffer = realloc(state->pending.pending_writes_buffer,
sizeof(uv_buf_t) * state->pending.pending_writes_count);

state->pending.pending_writes_buffer[state->pending.pending_writes_count - 1] = copy;

maybe_flush_ssl(state);

return 1;
}

This is fairly straightforward. We try to write to the buffer, and if we are successful, great. The  check_if_need_to_flush_ssl_state() will take care of actually sending that to the client.

If the write buffer is full, we empty it and try again. The interesting thing that happens is when we need to read in order to complete this write. In this case, we copy the data to write and store it on the side, then we proceed normally and wait or the libuv to deliver the next read buffer for this connection. When that is done, we’ll be sending the deferred write to the client.

It may be easier to explain the flow with a real example. When a new connection comes into the server, we create a new SSL context and then we call:

onnection_write(connection, "OK\r\n", 4);

This is the very first time that we actually interact with the SSL instance, and the call to SSL_write() is going to fail (because we haven’t established the SSL connection) with a SSL_ERROR_WANT_READ message. In response for this, we’ll copy the buffer we got and place it into the pending_writes of this connection. We also start listening to new data on the connection. The client will send the ClientHello message, which we’ll read and then feed into the SSL instance. That will cause us to write the SeverHello to the in memory buffer. When the  check_if_need_to_flush_ssl_state() is called, it will flush that message to the client.

Eventually, we’ll get the connection established, and at this point, we’ll be sending the deferred write to the client.

There are a bunch of other details, but they aren’t crucial to understanding this approaching. You can find the whole code sample here. I’ll reiterate again that it doesn’t have proper error handling, but it is less than 350 lines of C code that does something that is quite nice and expose an API that should be quite interesting to consume.

I’m really interested in feedback on this blog post, both on whether this approach makes any sense and what you think about the code. Please let me know in the comments below!