20 Days of DynamoDB
A DynamoDB tip per day keeps the …? Find out with these twenty DynamoDB quick tips and examples accompanied by code snippets.
Join the DZone community and get the full member experience.
Join For FreeFor the next 20 days (don’t ask me why I chose that number), I will be publishing a DynamoDB quick tip per day with code snippets. The examples use the DynamoDB packages from AWS SDK for Go V2 but should be applicable to other languages as well.
Day 20: Converting Between Go and DynamoDB Types
Posted: 13/Feb/2024
The DynamoDB attributevalue
in the AWS SDK for Go package can save you a lot of time, thanks to the Marshal
and Unmarshal
family of utility functions that can be used to convert between Go types (including struct
s) and AttributeValue
s.
Here is an example using a Go struct
:
MarshalMap
converts Customerstruct
into amap[string]types.AttributeValue
that's required byPutItem
UnmarshalMap
converts themap[string]types.AttributeValue
returned byGetItem
into a Customerstruct
type Customer struct {
Email string `dynamodbav:"email"`
Age int `dynamodbav:"age,omitempty"`
City string `dynamodbav:"city"`
}
customer := Customer{Email: "abhirockzz@gmail.com", City: "New Delhi"}
item, _ := attributevalue.MarshalMap(customer)
client.PutItem(context.Background(), &dynamodb.PutItemInput{
TableName: aws.String(tableName),
Item: item,
})
resp, _ := client.GetItem(context.Background(), &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{"email": &types.AttributeValueMemberS{Value: "abhirockzz@gmail.com"}},
})
var cust Customer
attributevalue.UnmarshalMap(resp.Item, &cust)
log.Println("item info:", cust.Email, cust.City)
Recommended reading:
- MarshalMap API doc
- UnmarshalMap API doc
- AttributeValue API doc
Day 19: PartiQL Batch Operations
Posted: 12/Feb/2024
You can use batched operations with PartiQL as well, thanks to BatchExecuteStatement
. It allows you to batch reads as well as write requests.
Here is an example (note that you cannot mix both reads and writes in a single batch):
//read statements
client.BatchExecuteStatement(context.Background(), &dynamodb.BatchExecuteStatementInput{
Statements: []types.BatchStatementRequest{
{
Statement: aws.String("SELECT * FROM url_metadata where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "abcd1234"},
},
},
{
Statement: aws.String("SELECT * FROM url_metadata where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "qwer4321"},
},
},
},
})
//separate batch for write statements
client.BatchExecuteStatement(context.Background(), &dynamodb.BatchExecuteStatementInput{
Statements: []types.BatchStatementRequest{
{
Statement: aws.String("INSERT INTO url_metadata value {'longurl':?,'shortcode':?, 'active': true}"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "https://github.com/abhirockzz"},
&types.AttributeValueMemberS{Value: uuid.New().String()[:8]},
},
},
{
Statement: aws.String("UPDATE url_metadata SET active=? where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberBOOL{Value: false},
&types.AttributeValueMemberS{Value: "abcd1234"},
},
},
{
Statement: aws.String("DELETE FROM url_metadata where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "qwer4321"},
},
},
},
})
Just like BatchWriteItem
, BatchExecuteStatement
is limited to 25 statements (operations) per batch.
Recommended reading:
BatchExecuteStatement
API docs- Build faster with Amazon DynamoDB and PartiQL: SQL-compatible operations (thanks, Pete Naylor !)
Day 18: Using a SQL-Compatible Query Language
Posted: 6/Feb/2024
DynamoDB supports PartiQL to execute SQL-like select, insert, update, and delete operations.
Here is an example of how you would use PartiQL-based queries for a simple URL shortener application. Notice how it uses a (generic) ExecuteStatement
API to execute INSERT
, SELECT
, UPDATE
and DELETE
:
_, err := client.ExecuteStatement(context.Background(), &dynamodb.ExecuteStatementInput{
Statement: aws.String("INSERT INTO url_metadata value {'longurl':?,'shortcode':?, 'active': true}"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "https://github.com/abhirockzz"},
&types.AttributeValueMemberS{Value: uuid.New().String()[:8]},
},
})
_, err := client.ExecuteStatement(context.Background(), &dynamodb.ExecuteStatementInput{
Statement: aws.String("SELECT * FROM url_metadata where shortcode=? AND active=true"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "abcd1234"},
},
})
_, err := client.ExecuteStatement(context.Background(), &dynamodb.ExecuteStatementInput{
Statement: aws.String("UPDATE url_metadata SET active=? where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberBOOL{Value: false},
&types.AttributeValueMemberS{Value: "abcd1234"},
},
})
_, err := client.ExecuteStatement(context.Background(), &dynamodb.ExecuteStatementInput{
Statement: aws.String("DELETE FROM url_metadata where shortcode=?"),
Parameters: []types.AttributeValue{
&types.AttributeValueMemberS{Value: "abcd1234"},
},
})
Recommended reading:
- Amazon DynamoDB documentation on PartiQL support
- ExecuteStatement API docs
Day 17: BatchGetItem
Operation
Posted: 5/Feb/2024
You can club multiple (up to 100) GetItem
requests in a single BatchGetItem
operation - this can be done across multiple tables.
Here is an example that fetches includes four GetItem
calls across two different tables:
resp, err := client.BatchGetItem(context.Background(), &dynamodb.BatchGetItemInput{
RequestItems: map[string]types.KeysAndAttributes{
"customer": types.KeysAndAttributes{
Keys: []map[string]types.AttributeValue{
{
"email": &types.AttributeValueMemberS{Value: "c1@foo.com"},
},
{
"email": &types.AttributeValueMemberS{Value: "c2@foo.com"},
},
},
},
"Thread": types.KeysAndAttributes{
Keys: []map[string]types.AttributeValue{
{
"ForumName": &types.AttributeValueMemberS{Value: "Amazon DynamoDB"},
"Subject": &types.AttributeValueMemberS{Value: "DynamoDB Thread 1"},
},
{
"ForumName": &types.AttributeValueMemberS{Value: "Amazon S3"},
"Subject": &types.AttributeValueMemberS{Value: "S3 Thread 1"},
},
},
ProjectionExpression: aws.String("Message"),
},
},
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
})
Just like an individual GetItem
call, you can include Projection Expressions and return RCUs. Note that BatchGetItem
can only retrieve up to 16 MB of data.
Recommended reading: BatchGetItem API doc
Day 16: Enhancing Write Performance With Batching
Posted: 2/Feb/2024
The DynamoDB BatchWriteItem
operation can provide a performance boost by allowing you to squeeze in 25 individual PutItem
and DeleteItem
requests in a single API call — this can be done across multiple tables.
Here is an example that combines PutItem
and DeleteItem
operations for two different tables (customer
, orders
):
_, err := client.BatchWriteItem(context.Background(), &dynamodb.BatchWriteItemInput{
RequestItems: map[string][]types.WriteRequest{
"customer": []types.WriteRequest{
{
PutRequest: &types.PutRequest{
Item: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: "c3@foo.com"},
},
},
},
{
DeleteRequest: &types.DeleteRequest{
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: "c1@foo.com"},
},
},
},
},
"orders": []types.WriteRequest{
{
PutRequest: &types.PutRequest{
Item: map[string]types.AttributeValue{
"order_id": &types.AttributeValueMemberS{Value: "oid_1234"},
},
},
},
{
DeleteRequest: &types.DeleteRequest{
Key: map[string]types.AttributeValue{
"order_id": &types.AttributeValueMemberS{Value: "oid_4321"},
},
},
},
},
},
})
Be aware of the following constraints:
- The total request size cannot exceed 16 MB
BatchWriteItem
cannot update items
Recommended reading: BatchWriteItem API doc
Day 15: Using the DynamoDB Expression Package To Build Update Expressions
Posted: 31/Jan/2024
The DynamoDB Go, SDK expression package, supports the programmatic creation of Update expressions.
Here is an example of how you can build an expression to include execute a SET
operation of the UpdateItem
API and combine it with a Condition
expression (update criteria):
updateExpressionBuilder := expression.Set(expression.Name("category"), expression.Value("standard"))
conditionExpressionBuilder := expression.AttributeNotExists(expression.Name("account_locked"))
expr, _ := expression.NewBuilder().
WithUpdate(updateExpressionBuilder).
WithCondition(conditionExpressionBuilder).
Build()
resp, err := client.UpdateItem(context.Background(), &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: "c1@foo.com"},
},
UpdateExpression: expr.Update(),
ConditionExpression: expr.Condition(),
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
ReturnValues: types.ReturnValueAllOld,
})
Recommended reading: WithUpdate method in the package API docs.
Day 14: Using the DynamoDB Expression Package To Build Key Condition and Filter Expressions
Posted: 30/Jan/2024
You can use expression
package in the AWS Go SDK for DynamoDB to programmatically build key condition and filter expressions and use them with Query
API.
Here is an example:
keyConditionBuilder := expression.Key("ForumName").Equal(expression.Value("Amazon DynamoDB"))
filterExpressionBuilder := expression.Name("Views").GreaterThanEqual(expression.Value(3))
expr, _ := expression.NewBuilder().
WithKeyCondition(keyConditionBuilder).
WithFilter(filterExpressionBuilder).
Build()
_, err := client.Query(context.Background(), &dynamodb.QueryInput{
TableName: aws.String("Thread"),
KeyConditionExpression: expr.KeyCondition(),
FilterExpression: expr.Filter(),
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
})
Recommended reading: Key and NameBuilder in the package API docs
Day 13: Using the DynamoDB Expression Package To Build Condition Expressions
Posted: 25/Jan/2024
Thanks to the expression
package in the AWS Go SDK for DynamoDB, you can programmatically build Condition expressions and use them with write operations.
Here is an example of the DeleteItem
API:
conditionExpressionBuilder := expression.Name("inactive_days").GreaterThanEqual(expression.Value(20))
conditionExpression, _ := expression.NewBuilder().WithCondition(conditionExpressionBuilder).Build()
_, err := client.DeleteItem(context.Background(), &dynamodb.DeleteItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
ConditionExpression: conditionExpression.Condition(),
ExpressionAttributeNames: conditionExpression.Names(),
ExpressionAttributeValues: conditionExpression.Values(),
})
Recommended reading: WithCondition method in the package API docs
Day 12: Using the DynamoDB Expression Package To Build Projection Expressions
Posted: 24/Jan/2024
The expression
package in the AWS Go SDK for DynamoDB provides a fluent builder API with types and functions to create expression strings programmatically along with corresponding expression attribute names and values.
Here is an example of how you would build a Projection Expression and use it with the GetItem
API:
projectionBuilder := expression.NamesList(expression.Name("first_name"), expression.Name("last_name"))
projectionExpression, _ := expression.NewBuilder().WithProjection(projectionBuilder).Build()
_, err := client.GetItem(context.Background(), &dynamodb.GetItemInput{
TableName: aws.String("customer"),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: "c1@foo.com"},
},
ProjectionExpression: projectionExpression.Projection(),
ExpressionAttributeNames: projectionExpression.Names(),
})
Recommended reading: expression package API docs.
Day 11: Using Pagination With Query API
Posted: 22/Jan/2024
The Query
API returns the result set size to 1 MB
. Use ExclusiveStartKey
and LastEvaluatedKey
elements to paginate over large result sets. You can also reduce page size by limiting the number of items in the result set with the Limit
parameter of the Query
operation.
func paginatedQuery(searchCriteria string, pageSize int32) {
currPage := 1
var exclusiveStartKey map[string]types.AttributeValue
for {
resp, _ := client.Query(context.Background(), &dynamodb.QueryInput{
TableName: aws.String(tableName),
KeyConditionExpression: aws.String("ForumName = :name"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":name": &types.AttributeValueMemberS{Value: searchCriteria},
},
Limit: aws.Int32(pageSize),
ExclusiveStartKey: exclusiveStartKey,
})
if resp.LastEvaluatedKey == nil {
return
}
currPage++
exclusiveStartKey = resp.LastEvaluatedKey
}
}
Recommended reading: Query Pagination
Day 10: Query API With Filter Expression
Posted: 19/Jan/2024
With the DynamoDB Query
API, you can use Filter Expressions to discard specific query results based on criteria. Note that the filter expression is applied after a Query finishes but before the results are returned. Thus, it has no impact on the RCUs (read capacity units) consumed by the query.
Here is an example that filters out forum discussion threads that have less than a specific number of views:
resp, err := client.Query(context.Background(), &dynamodb.QueryInput{
TableName: aws.String(tableName),
KeyConditionExpression: aws.String("ForumName = :name"),
FilterExpression: aws.String("#v >= :num"),
ExpressionAttributeNames: map[string]string{
"#v": "Views",
},
ExpressionAttributeValues: map[string]types.AttributeValue{
":name": &types.AttributeValueMemberS{Value: forumName},
":num": &types.AttributeValueMemberN{Value: numViews},
},
})
Recommended reading: Filter Expressions
Day 9: Query API
Posted: 18/Jan/2024
The Query
API is used to model one-to-many relationships in DynamoDB. You can search for items based on (composite) primary key values using Key Condition Expressions. The value for the partition key attribute is mandatory - the query returns all items with that partition key value. Additionally, you can also provide a sort key attribute and use a comparison operator to refine the search results.
With the Query
API, you can also:
- Switch to strongly consistent read (eventual consistent being the default)
- Use a projection expression to return only some attributes
- Return the consumed Read Capacity Units (RCU)
Here is an example that queries for a specific thread based on the forum name (partition key) and subject (sort key). It only returns the Message
attribute:
resp, err = client.Query(context.Background(), &dynamodb.QueryInput{
TableName: aws.String(tableName),
KeyConditionExpression: aws.String("ForumName = :name and Subject = :sub"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":name": &types.AttributeValueMemberS{Value: forumName},
":sub": &types.AttributeValueMemberS{Value: subject},
},
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
ConsistentRead: aws.Bool(true),
ProjectionExpression: aws.String("Message"),
})
Recommended reading:
Day 8: Conditional Delete Operation
Posted: 17/Jan/2024
All the DynamoDB write APIs, including DeleteItem
support criteria-based (conditional) execution. You can use DeleteItem
operation with a condition expression — it must be evaluated to true
in order for the operation to succeed.
Here is an example that verifies the value of inactive_days
attribute:
resp, err := client.DeleteItem(context.Background(), &dynamodb.DeleteItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
ConditionExpression: aws.String("inactive_days >= :val"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":val": &types.AttributeValueMemberN{Value: "20"},
},
})
if err != nil {
if strings.Contains(err.Error(), "ConditionalCheckFailedException") {
return
} else {
log.Fatal(err)
}
}
Recommended reading: Conditional deletes documentation
Day 7: DeleteItem
API
Posted: 16/Jan/2024
The DynamoDB DeleteItem
API does what it says - delete an item. But it can also:
- Return the content of the old item (at no additional cost)
- Return the consumed Write Capacity Units (WCU)
- Return the item attributes for an operation that failed a condition check (again, no additional cost)
- Retrieve statistics about item collections, if any, that were affected during the operation
Here is an example:
resp, err := client.DeleteItem(context.Background(), &dynamodb.DeleteItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
ReturnValues: types.ReturnValueAllOld,
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailureAllOld,
ReturnItemCollectionMetrics: types.ReturnItemCollectionMetricsSize,
})
Recommended reading: DeleteItem API doc
Day 6: Atomic Counters With UpdateItem
Posted: 15/Jan/2024
Need to implement an atomic counter using DynamoDB? If you have a use case that can tolerate over-counting or under-counting (for example, visitor count), use the UpdateItem
API.
Here is an example that uses the SET operator in an update expression to increment num_logins
attribute:
resp, err := client.UpdateItem(context.Background(), &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
UpdateExpression: aws.String("SET num_logins = num_logins + :num"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":num": &types.AttributeValueMemberN{
Value: num,
},
},
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
})
Note that every invocation of UpdateItem
will increment (or decrement) — hence, it is not idempotent.
Recommended reading: Atomic Counters
Day 5: Avoid Overwrites When Using DynamoDB UpdateItem
API
Posted: 12/Jan/2024
The UpdateItem
API creates a new item or modifies an existing item's attributes. If you want to avoid overwriting an existing attribute, make sure to use the SET
operation with if_not_exists
function.
Here is an example that sets the category of an item only if the item does not already have a category attribute:
resp, err := client.UpdateItem(context.Background(), &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
UpdateExpression: aws.String("SET category = if_not_exists(category, :category)"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":category": &types.AttributeValueMemberS{
Value: category,
},
},
})
Note that
if_not_exists
function can only be used in theSET
action of an update expression.
Recommended reading: DynamoDB documentation
Day 4: Conditional UpdateItem
Posted: 11/Jan/2024
Conditional operations are helpful in cases when you want a DynamoDB write operation (PutItem
, UpdateItem
or DeleteItem
) to be executed based on certain criteria. To do so, use a condition expression - it must evaluate to true in order for the operation to succeed.
Here is an example that demonstrates a conditional UpdateItem
operation. It uses the attribute_not_exists
function:
resp, err := client.UpdateItem(context.Background(), &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
UpdateExpression: aws.String("SET first_name = :fn"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":fn": &types.AttributeValueMemberS{
Value: firstName,
},
},
ConditionExpression: aws.String("attribute_not_exists(account_locked)"),
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
})
Recommended reading: ConditionExpressions
Day 3: UpdateItem Add-On Benefits
Posted: 10/Jan/2024
The DynamoDB UpdateItem
operation is quite flexible. In addition to using many types of operations, you can:
- Use multiple update expressions in a single statement
- Get the item attributes as they appear before or after they are successfully updated
- Understand which item attributes failed the condition check (no additional cost)
- Retrieve the consumed Write Capacity Units (WCU)
Here is an example (using AWS Go SDK v2):
resp, err = client.UpdateItem(context.Background(), &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
UpdateExpression: aws.String("SET last_name = :ln REMOVE category"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":ln": &types.AttributeValueMemberS{
Value: lastName,
},
},
ReturnValues: types.ReturnValueAllOld,
ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailureAllOld,
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
}
Recommended reading:
Day 2: GetItem
Add-On Benefits
Posted: 9/Jan/2024
Did you know that the DynamoDB GetItem
operation also gives you the ability to:
- Switch to strongly consistent read (eventually consistent being the default)
- Use a projection expression to return only some of the attributes
- Return the consumed Read Capacity Units (RCU)
Here is an example (DynamoDB Go SDK):
resp, err := client.GetItem(context.Background(), &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
//email - partition key
"email": &types.AttributeValueMemberS{Value: email},
},
ConsistentRead: aws.Bool(true),
ProjectionExpression: aws.String("first_name, last_name"),
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
})
Recommended reading:
Day 1: Conditional PutItem
Posted: 8/Jan/2024
The DynamoDB PutItem
API overwrites the item in case an item with the same primary key already exists. To avoid (or work around) this behavior, use PutItem
with an additional condition.
Here is an example that uses the attribute_not_exists
function:
_, err := client.PutItem(context.Background(), &dynamodb.PutItemInput{
TableName: aws.String(tableName),
Item: map[string]types.AttributeValue{
"email": &types.AttributeValueMemberS{Value: email},
},
ConditionExpression: aws.String("attribute_not_exists(email)"),
ReturnConsumedCapacity: types.ReturnConsumedCapacityTotal,
ReturnValues: types.ReturnValueAllOld,
ReturnItemCollectionMetrics: types.ReturnItemCollectionMetricsSize,
})
if err != nil {
if strings.Contains(err.Error(), "ConditionalCheckFailedException") {
log.Println("failed pre-condition check")
return
} else {
log.Fatal(err)
}
}
With the PutItem
operation, you can also:
- Return the consumed Write Capacity Units (WCU)
- Get the item attributes as they appeared before (in case they were updated during the operation)
- Retrieve statistics about item collections, if any, that were modified during the operation
Recommended reading:
Published at DZone with permission of Abhishek Gupta, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments