How To Handle Type Conversions With the DynamoDB Go SDK

DynamoDB provides a rich set of data types including Strings, Numbers, Sets, Lists, Maps etc. In the Go SDK for DynamoDB, the types package contains Go representations of these data types and the attributevalue module provides functions to work with Go and DynamoDB types.

This blog post will demonstrate how to handle conversions between Go types in your application and DynamoDB. We will start off with simple code snippets to introduce some of the API constructs and wrap up with an example of how to use these Go SDK features in the context of a complete application (including a code walk though).

You can refer to the complete code on GitHub.

To begin with, go through a few examples.

Please note: Error handling has been purposely omitted in the below code snippets to keep them concise.

Converting From Go to DynamoDB types

The Marshal family of functions takes care of this. It works with basic scalars (int, uint, float, bool, string), maps, slices, and structs.

To work with scalar types, just use the (generic) Marshal function:

func marshalScalars() {
    av, err := attributevalue.Marshal("foo")
    log.Println(av.(*types.AttributeValueMemberS).Value)

    av, err = attributevalue.Marshal(true)
    log.Println(av.(*types.AttributeValueMemberBOOL).Value)

    av, err = attributevalue.Marshal(42)
    log.Println(av.(*types.AttributeValueMemberN).Value)

    av, err = attributevalue.Marshal(42.42)
    log.Println(av.(*types.AttributeValueMemberN).Value)
}

Marshal converts a Go data type into an AttributeValue. But AttributeValue itself is just an interface and requires you to cast it to a concrete type such as AttributeValueMemberS (for string), AttributeValueMemberBOOL (for boolean) etc.

If you try to cast incompatible types, the SDK responds with a helpful error message. For example, panic: interface conversion: types.AttributeValue is *types.AttributeValueMemberN, not *types.AttributeValueMemberS. 

When working with slices and maps, you are better off using specific functions such as MarshalList and MarshalMap:

func marshalSlicesAndMaps() {
    avl, err := attributevalue.MarshalList([]string{"foo", "bar"})

    for _, v := range avl {
        log.Println(v.(*types.AttributeValueMemberS).Value)
    }

    avm, err := attributevalue.MarshalMap(map[string]interface{}{"foo": "bar", "boo": "42"})

    for k, v := range avm {
        log.Println(k, "=", v.(*types.AttributeValueMemberS).Value)
    }
}

The above examples gave you a sense of how to work with simple data types in isolation. In a real-world application, you will make use of composite data types to represent your domain model - most likely they will be in the form of Go structs. So let's look at a few examples of that.

Working With Go Structs

Here is a simple one:

type User struct {
    Name string
    Age  string
}

func marshalStruct() {
    user := User{Name: "foo", Age: "42"}

    av, err := attributevalue.Marshal(user)

    avm := av.(*types.AttributeValueMemberM).Value
    log.Println("name", avm["Name"].(*types.AttributeValueMemberS).Value)
    log.Println("age", avm["Age"].(*types.AttributeValueMemberS).Value)

    avMap, err := attributevalue.MarshalMap(user)

    for name, value := range avMap {
        log.Println(name, "=", value.(*types.AttributeValueMemberS).Value)
    }
}

Notice how convenient it is to use MarshalMap (instead of Marshal) when dealing Go structs especially if your application does not know all the attribute names.

So far, it seems like we can handle simple use cases, but we can do better. This example had a homogenous data type; i.e., the struct had only string type making it easy to iterate over the result map and cast the value to a *types.AttributeValueMemberS. If that were not the case, you would have to iterate over each and every attribute value type and typecast it to the appropriate Go type. This will be evident when working with the rest of the DynamoDB APIs. For example, the result of a GetItem invocation (GetItemOutput) contains a map[string]types.AttributeValue.

The SDK provides a way for us to make this much easier!

Converting From DynamoDB To Go Types

The Unmarshal family of functions takes care of this. Here is another example:

type AdvancedUser struct {
    Name         string
    Age          int
    IsOnline     bool
    Favourites   []string
    Contact      map[string]string
    RegisteredOn time.Time
}

func marshalUnmarshal() {
    user := AdvancedUser{
        Name:         "abhishek",
        Age:          35,
        IsOnline:     false,
        Favourites:   []string{"Lost In Translation, The Walking Dead"},
        Contact:      map[string]string{"mobile": "+919718861200", "email": "abhirockzz@gmail.com"},
        RegisteredOn: time.Now(),
    }

    avMap, err := attributevalue.MarshalMap(user)

    var result AdvancedUser
    err = attributevalue.UnmarshalMap(avMap, &result)

    log.Println("\nname", result.Name, "\nage", result.Age, "\nfavs", result.Favourites)
}

With MarshalMap, we converted an instance of AdvancedUser struct into a map[string]types.AttributeValue (imagine you get this as a response to a GetItem API call). Now, instead of iterating over individual AttributeValues, we simply use UnmarshalMap to convert it back a Go struct.

There is more! Utility functions like UnmarshalListOfMaps make it convenient to work with multiple slices of Go structs.

func marshalUnmarshalMultiple() {
    user1 := User{Name: "user1", Age: "42"}

    user1Map, err := attributevalue.MarshalMap(user1)
    if err != nil {
        log.Fatal(err)
    }

    user2 := User{Name: "user2", Age: "24"}

    user2Map, err := attributevalue.MarshalMap(user2)
    if err != nil {
        log.Fatal(err)
    }

    var users []User

    err = attributevalue.UnmarshalListOfMaps([]map[string]types.AttributeValue{user1Map, user2Map}, &users)
    if err != nil {
        log.Fatal(err)
    }

    for _, user := range users {
        log.Println("name", user.Name, "age", user.Age)
    }
}

Using struct Tags for Customization

Marshal and Unmarshal functions support the dynamodbav struct tag to control conversion between Go types and DynamoDB AttributeValue. Consider the following struct:

type User struct {
    Email string `dynamodbav:"email" json:"user_email"`
    Age   int    `dynamodbav:"age,omitempty" json:"age,omitempty"`
    City  string `dynamodbav:"city" json:"city"`
}

Here are a couple of common scenarios where the dynamodbav comes in handy:

Customize Attribute Name

Say we have a table with email as the partition key. Without the dynamodbav:"email" tag, when we marshal the User struct and try to save in the table, it will use Email (upper case) as the attribute name. DynamoDB will not accept this since attribute names are case sensitive: "All names must be encoded using UTF-8, and are case-sensitive."

Notice that we have combined JSON tags as well (this is perfectly valid): it's not used by DynamoDB, but the JSON library while encoding and decoding data.

Handle Missing Attributes

DynamoDB is a NoSQL database and tables don't have a fixed schema (except for partition key and an optional sort key). For example, a user item may not include the age attribute. By using dynamodbav:"age,omitempty", if the Age field is missing, it won't be sent to DynamoDB (it will be ignored).

In the absence of this tag, our DynamoDB record will have Age attribute set to 0. Depending on your use case this may or may not occur.

To look at all the usage patterns of this struct tag, refer to the Marshal API documentation.

As promised before, let's explore how to put all these APIs to use within the scope of an...

End-To-End Example

We will look at a Go application that exposes a REST API with a few endpoints. It combines the CRUD APIs (PutItem, GetItem, etc.) together with all the functions/APIs mentioned above.

Test the Application

Before we see the code, let's quickly review and test the endpoints exposed by the application. You will need to have Go installed, clone the application, and change to the right directory.

git clone https://github.com/abhirockzz/dynamodb-go-sdk-type-conversion
cd dynamodb-go-sdk-type-conversion

First, create a DynamoDB table (you can name it users). Use city as the Partition key, email as the Sort key.

You need some test data. You can do so manually, but I have included a simple utility to seed some test data during application startup. To use it, simply set the SEED_TEST_DATA variable at application startup:

export SEED_TEST_DATA=true

go run main.go

# output
started http server...

This will create 100 items. Check DynamoDB table to confirm:

Your application should be available at port 8080. You can use curl or any other HTTP client to invoke the endpoints:

# to get all users
curl -i http://localhost:8080/users/

# to get all users in a particular city
curl -i http://localhost:8080/users/London

# to get a specific user
curl -i "http://localhost:8080/user?city=London&email=user11@foo.com"

To better understand how the above APIs are used, let's briefly review key parts of the code:

Code Walkthrough

Add New Item to a DynamoDB Table

Starting with the HTTP handler for adding a User:

func (h Handler) CreateUser(rw http.ResponseWriter, req *http.Request) {
    var user model.User

    err := json.NewDecoder(req.Body).Decode(&user)
    if err != nil {// handle error}

    err = h.d.Save(user)
    if err != nil {// handle error}

    err = json.NewEncoder(rw).Encode(user.Email)
    if err != nil {// handle error}
}

First, we convert the JSON payload into a User struct which we then pass to the Save function.

func (d DB) Save(user model.User) error {

    item, err := attributevalue.MarshalMap(user)

    if err != nil {// handle error}

    _, err = d.client.PutItem(context.Background(), &dynamodb.PutItemInput{
        TableName: aws.String(d.table),
        Item:      item})

    if err != nil {// handle error}

    return nil
}

Notice how MarshalMap is used to convert the User struct to a map[string]types.AttributeValue that the PutItem API can accept:

Get a Single Item From DynamoDB

Since our table has a composite primary key (city is the partition key and email is the sort key), we will need to provide both of them to locate a specific user item:

func (h Handler) FetchUser(rw http.ResponseWriter, req *http.Request) {

    email := req.URL.Query().Get("email")
    city := req.URL.Query().Get("city")

    log.Println("getting user with email", email, "in city", city)

    user, err := h.d.GetOne(email, city)
    if err != nil {// handle error}


    err = json.NewEncoder(rw).Encode(user)
    if err != nil {// handle error}
}

We extract the email and city from the query parameters in the HTTP request and pass it on to the database layer (GetOne function).

func (d DB) GetOne(email, city string) (model.User, error) {

    result, err := d.client.GetItem(context.Background(),
        &dynamodb.GetItemInput{
            TableName: aws.String(d.table),
            Key: map[string]types.AttributeValue{
                "email": &types.AttributeValueMemberS{Value: email},
                "city":  &types.AttributeValueMemberS{Value: city}},
        })

    if err != nil {// handle error}

    if result.Item == nil {
        return model.User{}, ErrNotFound
    }

    var user model.User

    err = attributevalue.UnmarshalMap(result.Item, &user)
    if err != nil {// handle error}

    return user, nil
}

We invoke GetItem API and get back the result in form of a map[string]types.AttributeValue (via the Item attribute in GetItemOutput). This is converted back into the Go (User) struct using UnmarshalMap.

Notice: The Key attribute in GetItemInput also accepts a map[string]types.AttributeValue, but we don't use MarshalMap to create it.

Fetch Multiple Items

We can choose to query for all users in a specific city - this is a perfectly valid access pattern since city is the partition key.

The HTTP handler function accepts the city as a path parameter, which is passed on to the database layer.

func (h Handler) FetchUsers(rw http.ResponseWriter, req *http.Request) {
    city := mux.Vars(req)["city"]
    log.Println("city", city)

    log.Println("getting users in city", city)

    users, err := h.d.GetMany(city)

    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }

    err = json.NewEncoder(rw).Encode(users)
    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }
}

From there on, GetMany function does all the work:

func (d DB) GetMany(city string) ([]model.User, error) {

    kcb := expression.Key("city").Equal(expression.Value(city))
    kce, _ := expression.NewBuilder().WithKeyCondition(kcb).Build()

    result, err := d.client.Query(context.Background(), &dynamodb.QueryInput{
        TableName:                 aws.String(d.table),
        KeyConditionExpression:    kce.KeyCondition(),
        ExpressionAttributeNames:  kce.Names(),
        ExpressionAttributeValues: kce.Values(),
    })

    if err != nil {
        log.Println("Query failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    if len(result.Items) == 0 {
        return users, nil
    }

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)
    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

Pay attention to two things:

• How a KeyConditionExpression is being used (this is from the expressions package).
• More interestingly, the usage of UnmarshalListOfMaps function to directly convert a []map[string]types.AttributeValue (slice of items from DynamoDB) into a slice of User struct: If not for this function, we would need to extract each item from the result; i.e., a map[string]types.AttributeValue and call UnmarshalMap for each of them. So this is pretty handy!

Finally: Just Get Everything!

The GetAll function uses Scan operation to retrieve all the records in the DynamoDB table.

A Scan operation goes over the entire table (or secondary index) and it's highly likely that it will end up consuming a large chunk of the provisioned throughput, especially if it's a large table. It should be your last resort: check whether Query API (or BatchGetItem) works for your use case.

func (d DB) GetAll() ([]model.User, error) {

    result, err := d.client.Scan(context.Background(), &dynamodb.ScanInput{
        TableName: aws.String(d.table),
    })

    if err != nil {
        log.Println("Scan failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)

    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

Wrap Up

I hope you found this useful and now you are aware of the APIs in DynamoDB Go SDK to work with simple Go types as well as structs, maps, slices, etc. I would encourage you to explore some of the other nuances such as how to customize the Marshal and Unmarshal features using MarshalWithOptions and UnmarshalWithOptions respectively.