Use AWS Lambda and API Gateway With Node.js and Couchbase NoSQL

There has been a lot of buzz around functions as a service (FaaS), commonly referred to as serverless. A popular provider for these functions is Amazon with its AWS Lambda service. One could create a function in any of the supported Lambda technologies, let's say Node.js, deploy it to AWS Lambda, access it through the AWS API Gateway, and have it scale on its own to meet demand.

We're going to see how to use Node.js with AWS Lambda to create functions that can communicate with NoSQL documents in the flavor of Couchbase Server.

The assumption going forward is that you have a cluster or single instance of Couchbase Server running somewhere in the cloud. In other words, this project will not work if you're running Couchbase on your local machine. Because we're running this on AWS, it may make sense for Couchbase to exist as an EC2 instance. This can be done by using an available AMI.

Creating a FaaS Project for Lambda and API Gateway With Serverless Framework

If you've ever worked with AWS Lambda and AWS API Gateway before, you'll know that it isn't the most pleasant experience when starting something from scratch, then deploying. To make our lives a little easier, we're going to be using the Serverless Framework, which will take care of a lot of the heavy lifting, without changing how we develop Lambda functions.

Assuming you already have Node.js installed, execute the following to get the Serverless Framework CLI:

npm install -g serverless

With the CLI, we can generate new projects based on available templates, not limited to AWS. To create a new project with the CLI, execute the following:

serverless create --template aws-nodejs --path ./my-project

We'll be creating any number of functions within the my-project directory that was created. This is where things can get a little weird.

Our Node.js project is going to require the Couchbase SDK, a package for generating UUID values, and a package for data validation. The problem is that I'm using a Mac and Lambda is using a special flavor of Linux. If I try to download dependencies on my Mac, it will not be Linux compatible. Instead, we have to jump through a few hoops.

There is an article online titled Using Native Dependencies With AWS Lambda that explains how Docker can be used to accomplish the task of native project dependencies that work for Lambda.

In summary, you'd want to execute the following with Docker installed and with the CLI in your current project directory:

docker pull amazonlinux
docker run -v $(pwd):/lambda-project -it amazonlinux

The above commands would pull the Amazon Linux image and deploy a container, mapping the current project directory as a container volume. Within the container, Node.js can be installed and our package dependencies can be obtained. To see how to do this, check out a previous article that I wrote titled, Deploying Native Node.js Dependencies on AWS Lambda.

You'll need to install the following three dependencies for the project:

npm install couchbase --save
npm install joi --save
npm install uuid --save

Don't be deterred from continuing because of the Amazon Linux requirement. Essentially you're just running the three commands from within the container and since the container has a mapped volume, any files obtained will end up directly in your project.

Developing Functions for CRUD Operations Against the NoSQL Database

With the Serverless Framework project ready to go, we can focus on what really counts. The goal here is to create four different functions, one for each creating, retrieving, updating, and deleting Couchbase NoSQL documents. That's right, we're doing a CRUD set of functions.

Open the project's handler.js file because we need to bootstrap it:

'use strict';

const Couchbase = require("couchbase");
const UUID = require("uuid");
const Joi = require("joi");

var cluster = new Couchbase.Cluster("couchbase://your-server-here");
cluster.authenticate("demo", "123456");
var bucket = cluster.openBucket("example");

bucket.on("error", error => {
    console.dir(error);
});

module.exports.create = (event, context, callback) => { };

module.exports.retrieve = (event, context, callback) => { };

module.exports.update = (event, context, callback) => { };

module.exports.delete = (event, context, callback) => { };

In the above code, we've imported our project dependencies, connected to a Couchbase cluster, authenticated with said cluster, and opened a particular Bucket. Make sure to change the Couchbase information to reflect your actual information.

We've also created the four functions that were previously mentioned. However, as of now, these functions don't do anything.

Something important to note here. Our database connection logic is happening outside our functions. It is an expensive task to connect to the database with every invocation of a function. By moving this information outside, it is still accessible by any Lambda function containers.

Starting with the create function:

module.exports.create = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    var schema = Joi.object().keys({
        firstname: Joi.string().required(),
        lastname: Joi.string().required(),
        type: Joi.string().forbidden().default("person")
    });
    var data = JSON.parse(event.body);
    var response = {};
    var validation = Joi.validate(data, schema);
    if(validation.error) {
        response = {
            statusCode: 500,
            body: JSON.stringify(validation.error.details)
        };
        return callback(null, response);
    }
    var id = UUID.v4();
    bucket.insert(id, validation.value, (error, result) => {
        if(error) {
            response = {
                statusCode: 500,
                body: JSON.stringify({
                    code: error.code,
                    message: error.message
                })
            };
            return callback(null, response);
        }
        data.id = id;
        response = {
            statusCode: 200,
            body: JSON.stringify(data)
        };
        callback(null, response);
    });
};

There is a lot happening in the above code, so we need to break it down.

First, we're adding the following line:

context.callbackWaitsForEmptyEventLoop = false;

By adding the line above, we are changing how the function waits to respond. If we don't set it to false, the asynchronous things to follow will likely timeout before completion.

var schema = Joi.object().keys({
    firstname: Joi.string().required(),
    lastname: Joi.string().required(),
    type: Joi.string().forbidden().default("person")
});
var data = JSON.parse(event.body);
var response = {};
var validation = Joi.validate(data, schema);

Since we'll be creating data based on user input, it is a good idea to validate said user input. By using Joi, we can create a validation schema and use it to validate the data body passed in with the event. You might recognize that Joi is a validation framework that I've used in previous articles, such as, Create a RESTful API With Node.js, Hapi, and Couchbase NoSQL.

if(validation.error) {
    response = {
        statusCode: 500,
        body: JSON.stringify(validation.error.details)
    };
    return callback(null, response);
}

If there is a validation error, we'll send a 500 error back as a response. Otherwise we'll proceed in generating a new document key, and saving it to Couchbase Server:

var id = UUID.v4();
bucket.insert(id, validation.value, (error, result) => {
    if(error) {
        response = {
            statusCode: 500,
            body: JSON.stringify({
                code: error.code,
                message: error.message
            })
        };
        return callback(null, response);
    }
    data.id = id;
    response = {
        statusCode: 200,
        body: JSON.stringify(data)
    };
    callback(null, response);
});

Take note of how Lambda expects responses to be formatted. Responses should have a statusCode and a body.

Now, let's take a look at the retrieve function:

module.exports.retrieve = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    var response = {};
    var statement = "SELECT META().id, `" + bucket._name + "`.* FROM `" + bucket._name + "` WHERE type = 'person'";
    var query = Couchbase.N1qlQuery.fromString(statement);
    bucket.query(query, (error, result) => {
        if(error) {
            response = {
                statusCode: 500,
                body: JSON.stringify({
                    code: error.code,
                    message: error.message
                })
            };
            return callback(null, response);
        }
        response = {
            statusCode: 200,
            body: JSON.stringify(result)
        };
        callback(null, response);
    });
};

The goal of the retrieve function is to use N1QL to query for all documents in Couchbase and return them. Depending on the result, well format the Lambda response appropriately. Note we are altering the callbackWaitsForEmptyEventLoop value. This will happen in each of our functions.

How we structure the update function will be similar to how we structured the create function:

module.exports.update = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    var schema = Joi.object().keys({
        firstname: Joi.string().optional(),
        lastname: Joi.string().optional()
    });
    var data = JSON.parse(event.body);
    var response = {};
    var validation = Joi.validate(data, schema);
    if(validation.error) {
        response = {
            statusCode: 500,
            body: JSON.stringify(validation.error.details)
        };
        return callback(null, response);
    }
    var builder = bucket.mutateIn(event.pathParameters.id);
    if(data.firstname) {
        builder.replace("firstname", data.firstname);
    }
    if(data.lastname) {
        builder.replace("lastname", data.lastname);
    }
    builder.execute((error, result) => {
        if(error) {
            response = {
                statusCode: 500,
                body: JSON.stringify({
                    code: error.code,
                    message: error.message
                })
            };
            return callback(null, response);
        }
        response = {
            statusCode: 200,
            body: JSON.stringify(data)
        };
        callback(null, response);
    });
};

Notice that we start things off by doing some data validation based on our schema. If the data is considered valid, we can proceed to do subdocument operations against Couchbase:

var builder = bucket.mutateIn(event.pathParameters.id);
if(data.firstname) {
    builder.replace("firstname", data.firstname);
}
if(data.lastname) {
    builder.replace("lastname", data.lastname);
}
builder.execute((error, result) => {
    if(error) {
        response = {
            statusCode: 500,
            body: JSON.stringify({
                code: error.code,
                message: error.message
            })
        };
        return callback(null, response);
    }
    response = {
        statusCode: 200,
        body: JSON.stringify(data)
    };
    callback(null, response);
});

We're essentially creating a mutation builder based on the information found in the request. The mutations added will happen against a key when executed. The result of the execution will be returned as a response from Lambda.

Take note of the following:

event.pathParameters.id

This is data not coming from the body. It will be configured in one of the next steps.

Now we're left with the delete function:

module.exports.delete = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    var schema = Joi.object().keys({
        id: Joi.string().required()
    });
    var data = JSON.parse(event.body);
    var response = {};
    var validation = Joi.validate(data, schema);
    if(validation.error) {
        response = {
            statusCode: 500,
            body: JSON.stringify(validation.error.details)
        };
        return callback(null, response);
    }
    bucket.remove(data.id, (error, result) => {
        if(error) {
            response = {
                statusCode: 500,
                body: JSON.stringify({
                    code: error.code,
                    message: error.message
                })
            };
            return callback(null, response);
        }
        response = {
            statusCode: 200,
            body: JSON.stringify(data)
        };
        callback(null, response);
    });
};

If you've kept up with everything thus far, the delete function will look nothing new to you. We're validating data and executing it against the database.

With the functions created, we can focus on the Serverless Framework configuration information. Open the project's serverless.yml file and include the following:

service: couchbase-lambda

provider:
  name: aws
  runtime: nodejs6.10

functions:
  create:
    handler: handler.create
    events:
        - http:
            path: create
            method: post
  retrieve:
    handler: handler.retrieve
    events:
        - http:
            path: retrieve
            method: get
  update:
    handler: handler.update
    events:
        - http:
            path: update/{id}
            method: put
  delete:
    handler: handler.delete
    events:
        - http:
            path: delete
            method: delete

The most important part of the above YAML document is the functions. We've named each function and linked them to the appropriate handler function. The events are how these functions will be invoked from a browser. The events are configuration for AWS API Gateway.

In the events section, we define the path information, how each endpoint can be accessed (whether it be a GET request or a POST request or something else), and any kind of path variables.

Remember the line I told you to remember?

event.pathParameters.id

This is mapped to the id value in the update/{id} path.

At this point in time the project is ready to be deployed to Amazon Web Services.

Deploying the Functions to AWS Lambda With API Gateway Support

We won't get into the nitty-gritty when it comes to configuring Serverless Framework with your Amazon public and private keys, but we will worry about deploying, assuming those are configured.

From the Serverless Framework CLI, execute the following:

serverless deploy

The above command will take the functions code as well as the node_modules that were generated from the Amazon Linux Docker container, and push them to the AWS cloud. Serverless Framework will worry about configuring AWS Lambda and API Gateway based on what was in the serverless.yml file.

Conclusion

You just saw how to communicate with Couchbase Server from a set of functions controlled by AWS Lambda and AWS API Gateway. This is useful if you want to create applications that are billed for the time that they are used rather than the time that they are active. Including Couchbase wasn't much different than including it in any other Node.js application, with the exception of the native Node.js dependencies.

If you decide to use Couchbase in the cloud, make sure that the correct ports are open to Lambda. I made the error of forgetting to open all the ports that were required by the SDK and it set me back a bit.

For more information on using Couchbase Server, check out the Couchbase Developer Portal.