Writing a Microservice in Golang Which Communicates Over gRPC

This is the first installment of my three-part series on:

1. gRPC powered microservices in Golang.
2. Protecting Golang microservices with keycloak.
3. Writing java client to talk with our golang micro-service.

Let’s start!.

What is gRPC: gRPC stands for Google Remote Procedure Call. It’s a remote communication protocol created by Google which lets different services communicate with each other easily and efficiently. It offers synchronous and asynchronous communication to/from services. To learn more about gRPC, visit https://gRPC.io/

gRPC is best suitable for internal communications. It makes client calls much cleaner and we don’t need to worry about serialization, type safety and all those things as gRPC does this for us.

gPRC uses protobuf, a type-safe binary transfer format that is designed for efficient network communications. To learn more about protobuf, visit this link.

Performance benchmark results indicate that gRPC is a far better choice than HTTP/HTTP2 if performance and native call like experience is what the developer wants. You can find out more from the discussion here.

Building a Microservice in Golang

Let’s use the example of a Bitbucket repository creation flow: the user signs up for a Bitbucket account > selects the plan > creates a repository.

We want to create a service designed to handle various operations on the repository. When the user creates a repository, our service will be called to serve that operation. If the user is modifying repository settings, our service will be called to serve that operation. Basically, anything related to repository management will be served by our service.

We’ve chosen Golang (also referred to as Go) as our programming language for this service, gRPC as the communication protocol for other services to talk with our service, and keycloak to protect our service using the proven OpenId identity layer on OAuth 2.0 protocol.

Create the Message

To do this, first, we need to create a simple entity representation in gRPC called message. A message in gRPC terminology is something which can be used as a messag e(a message is defined using the protobuf syntax) to one service from another service. You can imagine a pigeon from one service carrying a message called “Winter has come” to another service and that service is consuming that message to carry out the contextual operation.

Now, in the above example, the service which sent the pigeon is the gRPC client, “Winter has come” is our message and service consuming the message is the gRPC server listening to that message. The nice thing about a message is that it can be transferred back and forth.

message Repository {
	int64 id  = 1;
	string name = 2;
	int64 userId = 3;
	bool isPrivate = 4;
}

Define the gRPC Service

Now that we have created a repository named message to be used in communication, the next step is to define the gRPC service.

service RepositoryService {
	//For now we’ll try to implement “insert” operation.
	rpc add (Repository) returns (AddRepositoryResponse);
}

message AddRepositoryResponse {
	Repository addedRepository = 1;
	Error error = 2;
}
message Error {
	string code = 1;
	string message = 2;
}

Here, we are telling the gRPC compiler that the piece of code starting with the “service” keyword should be treated as a gRPC service. The method preceded with the “rpc” keyword indicates that it’s a Remote Procedure Call and the compiler should generate appropriate stubs for the client and server runtimes.

We’ve also defined two more messages to tell our pigeon to return a success response or an error response after performing the operation.

Create a Folder Structure for the Golang Service

I am assuming you have a Go runtime set up. If you don’t, please follow the steps in their official documentation at https://golang.org/doc/install#install

We’ll also be using dep as our dependency management tool for our project. Dep is a mature solution for managing external dependencies in a Golang project. We use dep because Go module support is not yet officially released.

If you’re a Windows user, put the path to the dep installation in your environment’s PATH variable. This makes it easier as you’ll be able to use it without specifying the full path to the executable file.

Once Go runtime is installed, follow the below steps.

1. Create directory called “bitbucket-repository-management-service” at $GOPATH/src

Create these subpackages into that directory. The package layout I’m proposing is according to Go standards for package layout.

build
    → proto-gen.bat
    → proto-gen.sh
cmd
    → gRPC
        → server
            → main.go
        → client
            → main.go
Internal
    → gRPC
    → proto-files
        → domain
            → repository.proto
        → service
            → repository-service.proto
pkg

1. Navigate to the root directory of project and execute the below command:
   1. If Windows, “dep.exe init”
   2. If Linux, “dep init”
2. The above command will create a folder called “vendor” along with “Gopkg.lock” and “Gopkg.toml” files. These two files are important to manage different dependencies of our project.
3. Our next step is to place our proto files into the “internal” folder as these files are strictly bound to our application. Later, we’ll create a separate repository for this if we want to use the same files for different services in different programing languages. But, for simplicity, we’ll put those in the same directory for now.
4. Create folder called “proto-files” in the “internal” package as shown in the below image.
5. Inside the “proto-files”folder, create two sub folders
   1. domain
   2. service

So, the final project’s package layout will look like this.

Note: I am using VS Code for demonstration. Any other code editor or IDE is fine.

Next, we’ll paste the below code in the file called “repository.proto.” This code defines a skeleton message, written in protobuf syntax, which will be exchanged between the grpc client and server.

syntax = "proto3";

package domain;

option go_package = "bitbucket-repository-management-service/internal/gRPC/domain";

message Repository {
	int64 id  = 1;
	string name = 2;
	int64 userId = 3;
	bool isPrivate = 4;
}

After that, we’ll paste below code in the file called “repository-service.proto”. This code defines the grpc service definition. It defines the operation our grpc server will support along with possible input and return types.

syntax = "proto3";

package service;

option go_package = "bitbucket-repository-management-service/internal/gRPC/service";

import "bitbucket-repository-management-service/internal/proto-files/domain/repository.proto";

//RepositoryService Definition
service RepositoryService {
rpc add (domain.Repository) returns (AddRepositoryResponse);
}

message AddRepositoryResponse {
domain.Repository addedRepository = 1;
Error error = 2;
}
message Error {
string code = 1;
string message = 2;
}

Install gRPC compiler — The Great “PROTOC”

Without a gRPC compiler installed in our system, we won’t be able to generate stubs.

To install the protoc compiler,

1. navigate to this link,
2. Select latest release tag, make sure that you select a stable release.
3. Download the appropriate binary for your operating system.
4. Once downloaded, extract it to the location which is being scanned by the path variable of your operating system.

For windows,
“Environment Variables” → edit Path variable → add your folder location where you’ve extracted the downloaded binary.

For Linux family, 
Append to “/etc/profile” file to add extracted folder to PATH like this:

export PATH=$PATH:<<folder-location-to-protoc-binaries>>

The above steps will install the protoc compiler which we’ll use in a bit. There is one more thing we need to make available into our system → “Go bindings for protocol-buffers.”

Install Go Bindings and Generate Stubs

Without Go bindings, our stubs are useless. Go bindings provide helper structs, interfaces, and functions which we can use to register our gRPC service, marshal and unmarshal binary messages, etc.

To do that, we first need to add very simple Go code into our server.go file because, by defaul, dep (our dependency management tool) doesn’t download any libraries if there is no go code in the project.

To satisfy our beloved dep’s requirement, we’ll put some very basic go code in our cmd → gRPC → server → main.go file

package main

import "fmt"

func main() {
    fmt.Println("gRPC In Action!")
}

Now we are all good to install go bindings for proto buffers. We’ll execute the below command to install it.

Linux

dep ensure --add google.golang.org/gRPC github.com/golang/protobuf/protoc-gen-go

Windows

dep.exe ensure -add google.golang.org/gRPC github.com/golang/protobuf/protoc-gen-go

The above command will download go bindings into the “vendor” folder.

Now it’s time to generate stubs.

If you are on Windows, execute this command.

protoc.exe -I $env:GOPATH\src --go_out=$env:GOPATH\src $env:GOPATH\src\bitbucket-repository-management-service\internal\proto-files\domain\repository.proto

protoc.exe -I $env:GOPATH\src --go_out=plugins=gRPC:$env:GOPATH\src $env:GOPATH\src\bitbucket-repository-management-service\internal\proto-files\service\repository-service.proto

If you are on Linux, execute this command.

protoc -I $GOPATH/src --go_out=$GOPATH/src $GOPATH/src/bitbucket-repository-management-service/internal/proto-files/domain/repository.proto

protoc -I $GOPATH/src --go_out=plugins=gRPC:$GOPATH/src $GOPATH/src/bitbucket-repository-management-service/internal/proto-files/service/repository-service.proto

The above command will generate stubs, like those shown below, marked sub-directories.

Implement gRPC Service Stub

Next, we'll write our own implementation,

1. We will create a package called “impl” in the “internal → gRPC” directory.
2. We will create a struct called RepositoryServiceGrpcImpl.
3. Make sure that our struct implements all the gRPC stub methods.

So, as we know, our gRPC service has a method called add.We wrote its definition in our proto file earlier in the process.

rpc add (domain.Repository) returns (AddRepositoryResponse);

To implement its service contract, we’ll start by declaring a struct which will be responsible for RepositoryService’s implementation.

package impl

import (
	"bitbucket-repository-management-service/internal/gRPC/domain"
	"bitbucket-repository-management-service/internal/gRPC/service"
	"context"
	"log"
)

//RepositoryServiceGrpcImpl is a implementation of RepositoryService Grpc Service.
type RepositoryServiceGrpcImpl struct {
}

//NewRepositoryServiceGrpcImpl returns the pointer to the implementation.
func NewRepositoryServiceGrpcImpl() *RepositoryServiceGrpcImpl {
	return &RepositoryServiceGrpcImpl{}
}

//Add function implementation of gRPC Service.
func (serviceImpl *RepositoryServiceGrpcImpl) Add(ctx context.Context, in *domain.Repository) (*service.AddRepositoryResponse, error) {
	log.Println("Received request for adding repository with id " + strconv.FormatInt(in.Id, 10))

	//Logic to persist to database or storage.
	log.Println("Repository persisted to the storage")

	return &service.AddRepositoryResponse{
		AddedRepository: in,
		Error:           nil,
		}, nil
}

Now it’s time to write server configurations, port configurations and a minimal test-client which we can execute to verify the overall flow.

Let’s start with gRPC server first.

Configure gRPC Server

We’ll create an instance of a  RepositoryServiceGrpcImpl that will look like this:

repositoryServiceImpl := impl.NewRepositoryServiceGrpcImpl()

Next, we’ll create net.Listener:

func getNetListener(port uint) net.Listener {
	lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
		panic(fmt.Sprintf("failed to listen: %v", err))
	}

	return lis
}

Then, we’ll create the gRPC server.

gRPCServer := gRPC.NewServer()

We’ll register our service implementation to gRPC server.

service.RegisterRepositoryServiceServer(gRPCServer, repositoryServiceImpl)

We’ll bind  net.Listener and the gRPC server to let it communicate from a specified port.

// start the server
if err := gRPCServer.Serve(netListener); err != nil {
	log.Fatalf("failed to serve: %s", err)
}

If we wire up all the things, we’ll have something like this:

package main

import (
    "bitbucket-repository-management-service/internal/gRPC/impl"
    "bitbucket-repository-management-service/internal/gRPC/service"
    "fmt"
    "log"
    "net"

    "google.golang.org/gRPC"
)

func main() {
    netListener := getNetListener(7000)
    gRPCServer := gRPC.NewServer()

    repositoryServiceImpl := impl.NewRepositoryServiceGrpcImpl()
    service.RegisterRepositoryServiceServer(gRPCServer, repositoryServiceImpl)

    // start the server
    if err := gRPCServer.Serve(netListener); err != nil {
        log.Fatalf("failed to serve: %s", err)
    }

}

func getNetListener(port uint) net.Listener {
    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
        panic(fmt.Sprintf("failed to listen: %v", err))
    }

    return lis
}

Configure the gRPC Client

First, we’ll create a connection to the gRPC server.

serverAddress := "localhost:7000"
conn, e := gRPC.Dial(serverAddress, gRPC.WithInsecure())

We’ll pass that connection to the gRPC client.

client := service.NewRepositoryServiceClient(conn)

We’ll call our gRPC method.

client.Add(context.Background(), &repositoryModel);

If we wire up things here too, it’ll look like this:

package main

import (
"bitbucket-repository-management-service/internal/gRPC/domain"
"bitbucket-repository-management-service/internal/gRPC/service"
"context"
"fmt"

"google.golang.org/gRPC"
)

func main() {
	serverAddress := "localhost:7000"

	conn, e := gRPC.Dial(serverAddress, gRPC.WithInsecure())

	if e != nil {
		panic(e)
	}
	defer conn.Close()

	client := service.NewRepositoryServiceClient(conn)

	for i := range [10]int{} {
		repositoryModel := domain.Repository{
		Id:        int64(i),
		IsPrivate: true,
		Name:      string("Grpc-Demo"),
		UserId:    1245,
	}

	if responseMessage, e := client.Add(context.Background(), &repositoryModel); e != nil {
		panic(fmt.Sprintf("Was not able to insert Record %v", e))
  	} else {
		fmt.Println("Record Inserted..")
		fmt.Println(responseMessage)
		fmt.Println("=============================")
		}
	}
}

Test the Flow

To run the gRPC server, execute the below command from your project’s root directory.

go run .\cmd\gRPC\server\main.go

To run thee client (use new terminal window for this please):

go run .\cmd\gRPC\client\main.go

You should see something like this on the client’s standard output stream.

And on the server side:

For now, we are not persisting any of the requests to the proper storage engine.

Summary

We’ve created a minimal flow with best practices in mind for gRPC request to response. On one end, our gRPC server is listening and serving requests and on the other end the client is sending requests to the server. We are using our custom messages to pass to/from the gRPC server/client.

Our implementation above is synchronous. We have not yet addressed asynchronous and streaming of responses from the server. I’ll try to do that in my next post where I will talk about various features of gRPC protocols.

Till that time, Namaste.

You’ll find all the code hosted at this bitbucket repository.