Using TLS With Rust (Part 1)

The next interesting step in my Rust network protocol exercise is to implement TLS. I haven’t looked at that yet, so it is going to be interesting.

The first search for “Rust TLS” gives me the rustls project, which seems to provide a native Rust implementation. There are also native-tls, which uses the system TLS library and binding for OpenSSL. I’m not sure how trustworthy rustls is, but I’m building a network protocol that is meant as an exercise, so I don’t really care. The first thing that I wanted to write was pretty trivial. Just convert the current code to use TLS on the wire, nothing more.

I wrote the following code to set things up:

fn new(cert_path: &str, key_path: &str, listen_uri: &str) -> Result<Server, ConnectionError> {

    fn open_cert_file<F,T>(file: &str, method: F) -> Result<Vec<T>, ConnectionError> 
        where F : Fn(&mut io::BufRead) -> Result<Vec<T>, ()> {
        let certfile = fs::File::open(file)?;
        let mut reader = io::BufReader::new(certfile);
        match method(&mut reader) {
            Err(_) => return Err(ConnectionError::failed_cert(file)),
            Ok(certs) => match certs.len() {
                0 => return Err(ConnectionError::failed_cert(file)),
                _ => Ok(certs)
            }
        }
    }

    let mut config = rustls::ServerConfig::new(rustls::NoClientAuth::new());

    let certs = open_cert_file(cert_path, rustls::internal::pemfile::certs)?;
    let key = open_cert_file(key_path, rustls::internal::pemfile::rsa_private_keys)
                .or_else(|_| open_cert_file(key_path, rustls::internal::pemfile::pkcs8_private_keys))
                .and_then(|keys| match keys.get(0){
                    None => Err(ConnectionError::failed_cert(key_path)),
                    Some(key) => Ok(key.clone())
                })?;

    config.set_single_cert(certs, key)?;

    let listener = TcpListener::bind(listen_uri)?;

    Ok(Server { tls_config: Arc::new(config), tcp_listener: listener })
}

There is a lot going on and I spent some time figuring out exactly what needs to be done here. I’m kinda proud and also kind of scared of this code. It is packed. But on the other hand, if you read it, it is fairly clear — it just does a lot of stuff.

I then used this in the following way:

And that led to this error when using OpenSSL to connect to it:

On the Rust side, it died with:

The good thing is that both sides agree that there is a handshake issue, but I had no idea what was going on. The bad thing: I have no clue why this is failing. According to the docs, this is supposed to work. I tried debugging into rustls and it seems to be failing very early in the handshake process, but I’m not familiar enough with either Rust or TLS to really understand why. The core issues seem to be here:

Note that we get a payload of None, but when I’m debugging through the code in the read_version  method, it seems like it returns properly.

Oh, I figured it out; the issue is here:

Note the highlighted section? If this returns a None, it will silently return from the method. It is easy to miss something like that. Digging (much) deeper, I found:

And deeper still, we have:

Go back a bit and see what kind of URL I gave to OpenSSL; it was 127.0.0.1, and it seems like rustls doesn’t support raw IPs, only hostnames. The limit seems to be the result of this code:

This seems to be a known error that has been opened since May 2017, and it is a complete deal breaker for me. I’m guessing that this isn’t a big issue, in general, because usually if you have a cert, it is for a hostname and certificates for IPs (which exists but tend to be issued for private CAs) are far rarer.

I changed my cmd line to use:

And it started working, but at that point, I was debugging this issue for over an hour, so that is quite enough for today.

As an aside, I was able to attach to the Rust process using Visual Studio and debug it fairly normally. There is some weirdness that I believe relates to the cleanup where, if you abort a method early, it looks like it goes back in time until it exits the method. However, overall, it works fairly nicely. I do have to point out that many of the nicer features in the language are making debugging much harder.

That said, one thing was absolutely amazing — the fact that I so easily debugged into my dependencies. In fact, I was changing the code of my dependencies and re-running it to help me narrow down exactly where the error was. That was easy, obvious, and worked. Very nice!