Tutorial: Build DynamoDB-Compatible Apps for Any Cloud (Or On-Prem)

Alternator is an open-source project that allows teams to build "run-anywhere" DynamoDB-compatible applications with ScyllaDB.  This means you can deploy wherever you want: on-premises or on any public cloud.

In this tutorial, we’ll start by introducing the project. Afterward, we’ll see a hands-on example of creating a one-node ScyllaDB cluster and performing some basic operations on it.

The goal of this project is to deliver an open-source alternative to DynamoDB, deployable wherever a user would want: on-premises, on other public clouds like Microsoft Azure or Google Cloud Platform, or still on AWS (for users who wish to take advantage of other aspects of Amazon’s market-leading cloud ecosystem, such as the powerful i4 instances). DynamoDB users can keep their same client code unchanged. Alternator is written in C++ and is a part of ScyllaDB.

The three main benefits ScyllaDB Alternator provides to DynamoDB users are:

1. Cost: DynamoDB charges for read and write transactions (RCUs and WCUs). A free, open-source solution doesn’t.
2. Performance: ScyllaDB was implemented in modern C++. It supports advanced features that enable it to improve latency and throughput significantly.
3. Openness: ScyllaDB is open-source. It can run on any suitable server cluster regardless of location or deployment method.

Setting Up a ScyllaDB Cluster

If you haven’t done so yet, download the example from git:

git clone https://github.com/scylladb/scylla-code-samples.git

Go to the directory of the alternator example:

cd scylla-code-samples/alternator/getting-started

Next, we’ll start a one-node cluster with Alternator enabled.

By default, ScyllaDB does not listen to DynamoDB API requests. To enable such requests, we will set the alternator-port configuration option to the port (8000 in this example), which will listen for DynamoDB API requests.

Alternator uses ScyllaDB’s LWT feature. You can read more about it in the documentation.

docker run --name some-scylla --hostname some-scylla -p 8000:8000 -d scylladb/scylla:4.0.0 --smp 1 --memory=750M --overprovisioned 1 --alternator-port=8000

Wait a few seconds and make sure the cluster is up and running:

docker exec -it some-scylla nodetool status

[guy@localhost ~]$ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.2  79.82 KB   256          ?       1ccbb599-ce5c-4d72-9c3a-991e2567d0be  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
[guy@localhost ~]$ 

In this example, we will use the Python language to interact with ScyllaDB with the Boto 3 SDK for Python. It’s also possible to use the CLI or other languages such as Java, C#, Python, Perl, PHP, Ruby, Erlang, or JavaScript.

Next, if you don’t already have it set up, install the Boto 3 Python library which also contains drivers for DynamoDB:

sudo pip install --upgrade boto3

In the three scripts (create.py, read.py, and write.py) change the value for “endpoint_url” to the IP address of the node.

Create a Table

Now, we’ll use the create.py script to create a table in our newly created cluster, using Alternator.

from __future__ import print_function
import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://172.17.0.2:8000',
                  region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

table = dynamodb.create_table(
    BillingMode='PAY_PER_REQUEST',
    TableName='mutant_data',
    KeySchema=[
    {
        'AttributeName': 'last_name',
        'KeyType': 'HASH'
    },
    ],
    AttributeDefinitions=[
    {
        'AttributeName': 'last_name',
        'AttributeType': 'S'
    },
    ]
)

print("Finished creating table ", table.table_name ,". Status: ", table.table_status)

Authorization is not in the scope of this lesson, so we’ll use "None" and revisit this in a future lesson.

We define a table called "mutant_data" with the required properties such as the primary key “last_name” which is of a String data type. You can read about Boto 3 data types here.

The DynamoDB data model is similar to ScyllaDB’s. Both databases have a partition key (also called “hash key” in DynamoDB) and an optional clustering key (called “sort key” or “range key” in DynamoDB), and the same notions of rows (which DynamoDB calls “items”) inside partitions. There are some differences in the data model. One of them is that in DynamoDB, columns (called “attributes”), other than the hash key and sort key, can be of any type, and can be different in each row. That means they don’t have to be defined in advance. You can learn more about data modeling in Alternator in more advanced lessons.

In this simple example, we use a one-node ScyllaDB cluster. In a production environment, it’s recommended to run a cluster of at least three nodes.

Also, in this example, we’ll send the queries directly to our single node. In a production environment, you should use a mechanism to distribute different DynamoDB requests to different ScyllaDB nodes, to balance the load. More about that in future lessons.

Run the script:

python create.py

[guy@localhost getting-started]$ python create.py
Finished creating table  mutant_data . Status:  ACTIVE
[guy@localhost getting-started]

Each Alternator table is stored in its own keyspace, which ScyllaDB automatically creates. Table xyz will be in keyspace alternator_xyz. This keyspace is initialized when the first Alternator table is created (with a CreateTable request). The replication factor (RF) for this keyspace and all Alternator tables is chosen at that point, depending on the size of the cluster: RF=3 is used on clusters with three or more live nodes. RF=1 would be used if our cluster is smaller, as it is in our case. Using a ScyllaDB cluster of fewer than three nodes is not recommended for production.

Performing Basic Queries

Next, we will write and read some data from the newly created table.

from __future__ import print_function
import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://172.17.0.2:8000',
                  region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

response = dynamodb.batch_get_item(
    RequestItems={
        'mutant_data' : { 'Keys': [{ 'last_name': 'Loblaw' }, {'last_name': 'Jeffries'}] }
    }
)
for x in response:
    print (x)
    for y in response[x]:
        print (y,':',response[x][y])

In this script, we use the batch_write_item operation to write data to the table “mutant_data.” This allows us to write multiple items in one operation. Here we write two items using a PutRequest, which is a request to perform the PutItem operation on an item.

Notice that, unlike ScyllaDB (and Apache Cassandra for that matter), in DynamoDB, Writes do not have a configurable consistency level. They use CL=QUORUM.

Execute the script to write the two items to the table:

python write.py

[guy@localhost getting-started]$ python write.py
('Finished writing to table ', 'mutant_data')
[guy@localhost getting-started]$ 

Next, we’ll read the data we just wrote, again using a batch operation, batch_get_item.

from __future__ import print_function
import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://172.17.0.2:8000',
                  region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

response = dynamodb.batch_get_item(
    RequestItems={
        'mutant_data' : { 'Keys': [{ 'last_name': 'Loblaw' }, {'last_name': 'Jeffries'}] }
    }
)
for x in response:
    print (x)
    for y in response[x]:
        print (y,':',response[x][y])

The response is a dictionary with the result being the two entries we previously wrote.

Execute the read to see the results:

[guy@localhost getting-started]$ python read.py
Responses
mutant_data : [{u'first_name': u'Bob', u'last_name': u'Loblaw', u'address': u'1313 Mockingbird Lane'}, {u'first_name': u'Jim', u'last_name': u'Jeffries', u'address': u'1211 Hollywood Lane'}]
ResponseMetadata
RetryAttempts : 0
HTTPStatusCode : 200
HTTPHeaders : {'date': '19 Apr 2020 17:30:23 GMT', 'content-length': '219', 'content-type': 'application/json', 'server': 'Seastar httpd'}
[guy@localhost getting-started]$ 

DynamoDB supports two consistency levels for reads, “eventual consistency” and “strong consistency.” You can learn more about ScyllaDB consistency levels here and here. Under the hood, ScyllaDB implements Strongly-consistent reads with LOCAL_QUORUM, while eventually-consistent reads are performed with LOCAL_ONE.

More Resources

• Project Alternator Wiki, with examples (GitHub)
• ScyllaDB Cloud versus Amazon DynamoDB, benchmark
• Alternator Design Documentation

Conclusion

In this lesson, we learned the basics of Alternator, the open-source DynamoDB ScyllaDB API. We saw how to create a cluster, connect to it, write data, and read data.