How to Deploy Your Microservices Under Kubernetes With Gravitee and Elassandra

Gravitee.io is an open-source, flexible, lightweight and super-fast API Management solution that helps to control who, when and how consumers access your APIs. 

You may also like: Deploy a Spring Boot Microservice Architecture to Google Cloud and Google Kubernetes Engine

By default, the Gravitee API Manager relies on MongoDB for its configuration and Elasticsearch for log analytics, but deploying many components can be more tricky and more expensive than having a single distributed data store. To simplify the stack, the Gravitee repository for Elassandra can be used to store Gravitee configuration, Rate Limits, Logs, and Metrics, along with your application data, with the following benefits:

• Distributed Gravitee configuration on many datacenters through the Cassandra replication (inactive/active mode).

• Elasticsearch reporting for Gravitee analytics (Gravitee logs and metrics).
• Scalability by adding Elassandra nodes (without re-indexing) and data centers (for geolocalisation concerns or workload separation).
• Reduce the global complexity and TCO by using the same NoSQL datastore for both Gravitee and APIs data storage.

We will now see how to deploy all these components under Kubernetes.

Deploy Traefik

To expose the Gravitee components to the internet (gateways and management services), we use Traefik, the modern reverse proxy. Traefik watches for incoming kubernetes ingresses events and automatically deploy routers, services, handlers, etc.

Traefik is deployed using a HELM Chart as follow:

helm install --name traefik --namespace kube-system \

--set rbac.enabled=true,debug.enabled=true \

--set dashboard.enabled=true,dashboard.domain=traefik.${MY_DOMAIN} \

stable/traefik

Deploy an Elassandra Cluster

Three Elassandra pods have been deployed as a Kubernetes statefulset using this HELM chart.

helm upgrade --namespace default \
--set config.cluster_size=3 \
--set persistence.storageClass=managed-premium \
-f elassandra-enterprise-values.yaml \
ela elassandra

IMPORTANT NOTE: Cassandra spread replicas across racks to ensure data remains available if a rack goes down.  Deploying Cassandra or Elassandra with one Kubernetes Statefulset cannot guarantee that pods are located on at least three cloud-provider zones or regions. To meet this constraint, Strapdata will release soon a Kubernetes operator suitable for production.

Deploy Gravitee

As shown in the main figure, Gravitee has three components:

• The manager service to manage the Gravitee configuration.
• The user interface (or portal).
• The gateways, to manage access to your API endpoints.

To deploy Gravitee including the Elassandra repository, we have to use the HELM chart available at helm-Gravitee. In the HELM value.yaml, the following es, and elassandra blocks describe the connection to both Elasticsearch (for Gravitee analytics) and to Elassandra, which is very close to Apache Cassandra. The elassandra.endpoint specifies the Elasticsearch endpoint to be able to create and search across elasticsearch indices.

es:
  cluster: elassandra
  index: analytics
  # If the details for security are entered
  # authentication will be provided for the
  # elastic search cluster
  # https://docs.gravitee.io/apim_installguide_repositories_elasticsearch.html#management_api_configuration
  security:
    enabled: true
    username: cassandra
    password: cassandra
    ssl:
      truststore:
        path: /ca-pub/truststore.p12
        password: changeit
  endpoints:
    - https://dc1-elassandra-datacenter-elasticsearch.default.svc.cluster.local:9200

elassandra:
  clusterName: elassandra
  contactPoint: dc1-elassandra-datacenter.default.svc.cluster.local
  endpoint: https://dc1-elassandra-datacenter-elasticsearch.default.svc.cluster.local:9200
  port: 39042
  username: cassandra
  password: cassandra
  ssl:
    truststore:
      path: /ca-pub/truststore.p12
      password: changeit

Detail about deploying Gravitee under Kubernetes can be found here on Medium.

Deploy Your Application

We deploy here a sample Micronaut application basketapp allowing to store and search baskets stored as JSON documents through a REST API. The basketapp stores its data into Elassandra, using Cassandra CQL to insert/select rows, and Elasticsearch queries through the CQL driver to search for documents.

This basic application is deployed as a Kubernetes deployment with a HELM chart included in the basketapp code.

When everything is deployed, we have the following pods running in our Kubernetes cluster:

NAME                                                       READY   STATUS    RESTARTS   AGE
pod/basketapp-79cd9d8c95-zt89z                             1/1     Running   0          4m24s
pod/elassandra-cl1-dc1-0-0                                 2/2     Running   0          4h43m
pod/elassandra-cl1-dc1-1-0                                 2/2     Running   0          4h40m
pod/elassandra-cl1-dc1-2-0                                 2/2     Running   0          4h37m
pod/gravitee-api-fd6647f45-5fvhf                           1/1     Running   0          3h40m
pod/gravitee-gateway-755c7568c6-54z45                      1/1     Running   0          3h40m
pod/gravitee-ui-5bc87f599c-8fxjs                           1/1     Running   0          3h40m

The three Gravitee components (API management, UI, and gateway) are accessible from outside the kubernetes cluster through traefik with an external load balancer.

On an Elassandra pod, we can check that the basketapp and the Gravitee elasticsearch indices are green:

cassandra@dc1-elassandra-datacenter-0:/$ curl -s -XGET http://$localhost:9200/_cat/indices?v

health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   alerts                       mUmki6_0RUKuCgE-yu0Xwg   1   0          0            0       208b           208b
green  open   analytics-monitor-2019.07.23 u-lWGgnsTm2xawuTMK5tyQ   1   0       1834            0    449.4kb        449.4kb
green  open   apikeys                      Q5asfLdYRjSWr49LHtUkBQ   1   0          0            0       208b           208b
green  open   apis                         b2rpfxabStyiRIcwIgKbSQ   1   0          1            0     11.2kb         11.2kb
green  open   audits                       U3_qOxASQCqWomHccwaLaQ   1   0         59            0      113kb          113kb
green  open   baskets                      czegFp8VRgKkchujRFiBAw   1   0          4            0        5kb            5kb
green  open   commands                     tWYKgummTUWonw9uosd-QA   1   0          7            0     42.3kb         42.3kb
green  open   events                       jDj7XUSSRCK70blKqS8Rng   1   0          8            0     27.4kb         27.4kb
green  open   generic_notification_configs iYxCxkU6Th-iND6SKKgY4w   1   0          0            0       208b           208b
green  open   invitations                  _VezToE2QQiy-3ZmBZ1jqQ   1   0          0            0       208b           208b
green  open   media                        FetLDUONRDaYvDZx_7SKDw   1   0          0            0       208b           208b
green  open   memberships                  VyXePS1_QPO1Tij-A7bV_A   1   0          4            0     18.2kb         18.2kb
green  open   metadata                     1PwYbj3GTqeAFcQUv8fzXQ   1   0          1            0      4.7kb          4.7kb
green  open   pages                        1OtcBLKHRMONWKUBXT21yw   1   0          1            0      9.9kb          9.9kb
green  open   plans                        OXjusYNlT2m8OmwHFo0PSw   1   0          1            0      6.7kb          6.7kb
green  open   portal_notification_configs  nkYxdH2mQwusIsySG4fQlQ   1   0          0            0       208b           208b
green  open   portal_notifications         RC1hZooIQDyhmiXOgCSwPg   1   0          0            0       208b           208b
green  open   ratinganswers                uCK7DB76Ts6gOaMmR6oJNQ   1   0          0            0       208b           208b
green  open   ratings                      X6o6iuRbT_Sm5J4WxZmZWQ   1   0          0            0       208b           208b
green  open   subscriptions                qHG1qGgSSwWGgdNz3Asi-w   1   0          0            0       208b           208b
green  open   users                        h_TQsCrtRjK5dsevqnGHwA   1   0          1            0      4.7kb          4.7kb
green  open   workflows                    AMJSKc-AQCWDjIMOvF1Ztw   1   0          0            0       208b           208b

Register Your API Into Gravitee

As described in the Micronauts documentation, we use an annotation in the Micronaut Application.java, where the servers.url points to our application. Be careful, URL context path cannot be null:

@OpenAPIDefinition(
    info = @Info(
        title = "BasketApp",
        version = "0.2",
        description = "BasketApp REST API",
        license = @License(name = "Apache license", url = "https://github.com/strapdata/basketapp/LICENSE.txt")
    ),
    servers = @Server(
        url = "http://basketapp:8080/basketapp"
    ),
    tags = {
        @Tag(name = "basket")
    }
)
public class Application {
    public static void main(String[] args) {
        Micronaut.run(Application.class);
    }
}

The swagger descriptor is generated at compile time by Micronaut, and server URLs may depend on your deployments, so we introduced a controller to dynamically replace the servers.url with the application Kubernetes service name. The swagger descriptor is available from a pod at http://basketapp:8080/basketapp/swagger, the URL used to import the API into Gravitee:

1. Log in to the API Manager portal as the administrator and register your application by providing a swagger descriptor. 

2. Start and publish the API.

3. Create an API plan.

To keep the demo simple, you choose a keyless authentication, but Gravitee provides many ways to secure access to your API.

5. Publish to the API plan (click on the cloud) and deploy it to the gateways (click on the orange bar).

Finally, our application is now available through the Gravitee gateway.

You can request your basket API through the Gravitee gateway service:

curl -k "https://gateway.mydomain.com/basketapp/basket/search"
[ {
  "id" : "fe2ace53-69d6-4f11-93b1-e0fde68a95ce",
  "items" : [ {
    "product_qty" : 1,
    "amount_paid" : 1.0,
    "product_code" : "1"
  }, {
    "product_qty" : 2,
    "amount_paid" : 2.0,
    "product_code" : "2"
  }, {
    "product_qty" : 3,
    "amount_paid" : 3.0,
    "product_code" : "3"
  } ],
  "store_code" : "1",
  "basket_status" : "Finished",
  "processing_date" : "2019-03-05T08:06:03.133+0000"
}, {
  "id" : "fe2ace53-69d6-4f11-93b1-e0fde68a95cc",
  "items" : [ {
    "product_qty" : 1,
    "amount_paid" : 1.0,
    "product_code" : "1"
  }, {
    "product_qty" : 2,
    "amount_paid" : 2.0,
    "product_code" : "2"
  }, {
    "product_qty" : 3,
    "amount_paid" : 3.0,
    "product_code" : "3"
  } ],
  "store_code" : "1",
  "basket_status" : "Finished",
  "processing_date" : "2019-03-05T08:06:03.133+0000"
} ]

Logs and Metrics

Thanks to Elassandra, Gravitee analytics works like Elasticsearch. You can check API performances, application load, and enable API logging for a while.

Conclusion

This article shows how to use Elassandra to store both application data and Gravitee configuration and logs. Elassandra and Gravitee can scale up/down to achieve high throughput and high availability. In the next article, I will show how to deploy this configuration on several Kubernetes clusters located in several datacenters.

Further Reading

Microservices on Kubernetes