Go and Redis: Better Together

I recently presented this talk at the Conf42 Golang 2023 and I thought it might be a good idea to turn it into a blog post for folks who don't want to spend 40+ mins watching the talk (it's ok, I understand) or just staring at slides trying to imagine what I was saying.

So, here you go!

By the way, you are still welcome to watch the talk or download the slides! There are a lot of great talks that you can get from this playlist.

This talk was geared toward folks who are looking to get started with Redis and Go. Or perhaps you are already experienced with both these topics – in that case, it might be a good refresher!

To that extent, I had a very simple agenda.

• I started off by setting the context about Redis and Go.
• Provided an overview of the Go and Redis ecosystem, including the client options you’ve got
• Followed by some hands-on stuff
• Wrapped up with some tips/tricks and resources

I Love Redis and Go!

Since its release in 2009, it did not take Redis too long to win the hearts and minds of the developer community! As per DB-engines trends statistics, Redis has been topping the charts since 2013. And on Stack Overflow annual survey, it’s been voted the most loved database for 5 years in a row.

Go has become the language of the Cloud. It powers many cloud-native projects (apparently, 75% of CNCF projects are written in Go) including Docker, Kubernetes, Prometheus, Terraform, etc. In fact, there are many databases written in Go - like InfluxDB, etcd, Vitess, TiDB, etc.

Go also caters to a wide variety of general-purpose use cases:

• Web apps and APIs
• Data processing pipelines
• Infrastructure as code
• SRE/DevOps solutions
• Command line apps (this is a really popular one!)
• And more

No wonder Go has become such a popular programming language!

Now you might be thinking, "Hey, Go is down at the bottom." But if you notice carefully, it is the only statically-typed lang after Rust (of course there are C# and Kotlin down there as well), and this is from 2022. If you look at data from 2021 to 2018, you will notice that Go has maintained its top 5 spot.

Go and Redis have a few things in common, but to me, simplicity is the one that really stands out to me.

Simplicity

Redis is a key-value store, but the values can be any of these data structures that you see. These are all data structures that we as developers use every day - lists, sets, maps, etc. Redis just feels like an extension of these core programming language constructs.

With Go, this comes in various forms:

• Excellent tooling
• A comprehensive standard library
• Easy-to-use concurrency primitives

And sometimes it's in the form of not bloating the language with unnecessary features. To cite an example, it took a while for generics to be added to the language. Now, I am not trying to trick you into thinking that Go is simple; or for that matter, that any programming language is simple.

But with Go, the goal is to give you simple components to help build complex things and hide complexity behind a simple facade.

There are folks who have explained this in great detail (and much better than I can!). I would really encourage you to check out this talk by Rob Pike (and the slides), the co-creator of Go (it's from 2015, but still very much applicable to the essence and spirit of Go).

Redis 101

A quick intro to Redis (some of the key points):

• Data structure server: At its core, Redis is nothing but a key-value store, where the key can be string or even a binary. The important thing to note is that as far as the value is concerned, you can choose from a variety of data structures such as strings,hashes, lists, sets, etc.
• Redis is not just a cache: it’s a really solid messaging platform as well.
• HA: You can configure Redis to be highly available by setting up primary-replica replication, or take it a step further with Redis Cluster.
• Persistence: Redis is primarily in-memory but you can configure it to persist to disk as well. There are solutions like Amazon MemoryDB that can actually take it a notch further (thanks to its distributed transactional log).
• Since it's open-source and wildly popular, you can get offerings from pretty much every cloud provider – big or small, or even run it on Kubernetes, on cloud, on-prem, or hybrid mode. If you want to put Redis in production, you can rest assured that there is no dearth of options for you to run and operate it.

Redis Data Types

What you see here is a list of core data structures:

• A string seems really simple, but is quite powerful. They can be used for something as simple as storing a key-value pair to implementing advanced patterns like distributed locking and rate-limiting.
• A hash is similar to a map in Java, or dictionary in Python. It is used to store object-like structures like user profiles, customer info, etc.
• A set behaves like its mathematical version: it helps you maintain a unique set of items along with the ability to list them, count them, execute unions, intersections, and so on.
• Think of sorted sets like this big brother to a set - They make it possible to implement things like leaderboards, which is very useful in areas like gaming. For example, you can store player scores in a sorted set and when you need to get the top 10 players, you can simply invoke specific sorted set commands and get that data. The beauty is that sorting happens on the database side, so there is no client-side logic you need to apply.
• Lists are a really versatile data structure as well. You can use them to store many things, but using them as a worker queue is very common. There are popular open-source solutions such as sidekiq and celery that already support Redis as a backend for job queuing solutions.
• Redis streams (added in Redis 5) is used for streaming use-cases.
• Also, ephemeral messaging with pub/sub - it's a publish-broadcast mechanism where you can send/receive messages to/from channels.
• There is also Geospatial data structure and a really cool one called Hyperloglog which is an alternative to a traditional set. It can store millions of items while optimizing for data storage and you can count the number of unique items with really high accuracy.

Go Clients for Redis

These are the most popular Go clients for Redis.

go-redis is by far the most popular client. It has what you’d expect. Features, decent documentation, active community. Moving this under the official Redis GitHub org is just icing on the cake!

redigo is a fairly stable and tenured client that supports all the standard Redis data types and features such as transactions, pipelining, etc. It is also used to implement other Go client libraries such as redisearch-go and Redis TimeSeries Go client. That being said, its API is a bit too flexible. While some may prefer that, I feel that it’s not a good fit when I am using a type-safe language like Go (that’s just my personal opinion).

But the biggest drawback to me is that it does not support Redis Cluster!

rueidis is a relatively new (at the time of writing) but quickly evolving client library. It supports RESP3 protocol and client-side caching and supports a variety of Redis Modules. As far as the API is concerned, this client adopts an interesting approach. It provides a Do function as well (like redigo client), but the way it creates the command is via a builder pattern - this retains strong type checking (unlike redigo).

To be honest with you, I haven't used this client a lot, and it looks like it's packed with a lot of features. So, I can't complain much as of now!

For those who are looking for deeper performance numbers – this benchmark comparison with the go-redis library might be interesting.

Client Ecosystem

Now let's take a look from an ecosystem perspective. To clarify, the point here is not chest-thumping based on GitHub stars - it's just to give you a sense of things.

For folks not using Go and Redis, the popularity of the Go client might come as a surprise. Java workloads form a huge chunk of the Redis workloads and Jedis is the bread-and-butter client when it comes to Java apps with Redis (and it's pretty old). But I was surprised to see redisson topping the charts, followed by go-redis (yay!), (two) Node.js clients, Python, and finally back to Java.

Another thing to note is that I was looking for more than 10000 stars. At the time of writing, the phpredis client was close to 9600 stars.

Now, time for some practical stuff.

Demos

During the demo, I covered:

A walk-through of the Go Redis client:

• Basics such as connecting to Redis
• Using common data types like string with TTL
• Use hash and struct support in go-redis
• How to use set as well as pipelining technique
• Hyperloglog and how does it differ from a Set:

func hllandset() {
    pipe := client.Pipeline()

    ips := []string{}

    for i := 1; i <= 10_00_000; i++ {
        ips = append(ips, fake.IP(fake.WithIPv4()))
    }

    pipe.SAdd(context.Background(), "set_of_ips", ips)
    pipe.PFAdd(context.Background(), "hll_of_ips", ips)

    pipe.Exec(ctx)

    //redis-cli MEMORY usage set_of_ips
    fmt.Println("no. of unique views (SCARD) -", client.SCard(ctx, "set_of_ips").Val())

    //redis-cli MEMORY usage hll_of_ips
    fmt.Println("no. of unique views (PFCOUNT) -", client.PFCount(ctx, "hll_of_ips").Val())

• Chat application using pub/sub (below is a trimmed-down version of the code):

package main

import (
    //omitted
)

var client *redis.Client
var Users map[string]*websocket.Conn
var sub *redis.PubSub
var upgrader = websocket.Upgrader{}

const chatChannel = "chats"

func init() {
    Users = map[string]*websocket.Conn{}
}

func main() {
    //...connect to redis (omitted)

    broadcast()

    http.HandleFunc("/chat/", chat)
    server := http.Server{Addr: ":8080", Handler: nil}

    //...start server (omitted)

    exit := make(chan os.Signal, 1)
    signal.Notify(exit, syscall.SIGTERM, syscall.SIGINT)
    <-exit

    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    //... clean up all connected connections, unsubscribe and shutdown server (omitted)
}

func chat(w http.ResponseWriter, r *http.Request) {
    user := strings.TrimPrefix(r.URL.Path, "/chat/")

    upgrader.CheckOrigin = func(r *http.Request) bool {
        return true
    }

    // 1. create websocket connection
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Print("upgrade:", err)
        return
    }

    // 2. associate the user (name) with the actual connection
    Users[user] = c
    fmt.Println(user, "in chat")

    for {
        _, message, err := c.ReadMessage()
        if err != nil {
            //error handling and disconnect (omitted)
        }

        // 3. when a messages comes in via the connection, publish messages to redis channel
        client.Publish(context.Background(), chatChannel, user+":"+string(message))
    }
}

func broadcast() {
    go func() {
        sub = client.Subscribe(context.Background(), chatChannel)
        messages := sub.Channel()
        for message := range messages {
            from := strings.Split(message.Payload, ":")[0]
            // 3. if a messages is received on the redis channel, broadcast it to all connected sessions (users)
            for user, peer := range Users {
                if from != user {
                    peer.WriteMessage(websocket.TextMessage, []byte(message.Payload))
                }
            }
        }
    }()
}

Tips and Tricks

In this section, I covered some of the points from "Using Redis on Cloud? Here Are Ten Things You Should Know."

These include:

• Connecting to Redis - common mistakes
• Scalability options - Vertical, Horizontal
• Using read-replicas
• Influence how your keys are distributed across a Redis cluster
• Execute bulk operations across Redis Cluster
• Sharded Pub/Sub

Resources

Finally, I wrapped up with some resources, including the Discord channel for the Go Redis community.

Happy building!