Microservices Sidecar Pattern Implementation Using Postgres, Spring Cloud Netflix, and Docker

spring cloud series

1. developing microservices using spring boot, jersey, swagger and docker
2. integration testing using spring boot, postgres and docker
3. services registration and discovery using spring cloud netflix eureka server and client-side load-balancing using ribbon and feign
4. centralized and versioned configuration using spring cloud config server and git
5. routing requests and dynamically refreshing routes using spring cloud zuul server
6. microservices sidecar pattern implementation using postgres, spring cloud netflix and docker (you are here)
7. implementing circuit breaker using hystrix, dashboard using spring cloud turbine server (work in progress)

1. microservices sidecar pattern implementation using postgres, spring cloud netflix, and docker

what's a sidecar? a sidecar is a companion application of the main service, typically non-jvm, either developed in-house or a 3rd party service (eg elastic search, apache solr, etc.) where it's desirable for them to take advantage of other infrastructure services such as service registration and discovery , routing , dynamic configuration , monitoring, etc..

this post covers implementing a sidecar java application attached to a postgres database bundled in a docker image and a demo client application connecting to the database after retrieving the postgres metadata (e.g. host, port) from a eureka registry.

2. requirements

• java 7 or 8.
• maven 3.2+
• familiarity with spring framework.
• a eureka server instance for the spring cloud netflix sidecar application to register the bundled postgres server with.
• docker, local or remote host.

3. the sidecar application

can be created like any other spring cloud app, from your preferred ide, http://start.spring.io or from the command line:

curl "https://start.spring.io/starter.tgz"
 -d bootversion=1.5.9.release
 -d dependencies=actuator,cloud-eureka
 -d language=java
 -d type=maven-project
 -d basedir=sidecar
 -d groupid=com.asimio.cloud
 -d artifactid=sidecar
 -d version=0-snapshot
 | tar -xzvf -

this command will create a maven project in a folder named sidecar with most of the dependencies used in the accompanying source code for this post.

the sidecar relevant files are discussed next:

pom.xml:

...
<properties>
  ...
  <spring-cloud.version>dalston.sr4</spring-cloud.version>
</properties>

<dependencymanagement>
  <dependencies>
    <dependency>
      <groupid>org.springframework.cloud</groupid>
      <artifactid>spring-cloud-dependencies</artifactid>
      <version>${spring-cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencymanagement>

<dependencies>
  <dependency>
    <groupid>org.springframework.boot</groupid>
    <artifactid>spring-boot-starter-actuator</artifactid>
  </dependency>
  <dependency>
    <groupid>org.springframework.cloud</groupid>
    <artifactid>spring-cloud-starter-eureka</artifactid>
  </dependency>
  <dependency>
    <groupid>org.springframework.cloud</groupid>
    <artifactid>spring-cloud-netflix-sidecar</artifactid>
  </dependency>
...
</dependencies>

spring-cloud-starter-eureka includes eureka client support for applications to register and/or discovery services metadata with/from a remote eureka server.

spring-cloud-netflix-sidecar provides beans spring autoconfiguration for a companion application to take care of the service registration and/or discovery of 3rd party services (jvm or not).

sidecarapplication.java :

package com.asimio.cloud.sidecar;
...
@springbootapplication
@enablesidecar
public class sidecarapplication {

  public static void main(string[] args) {
    springapplication.run(sidecarapplication.class, args);
  }
}

@enablesidecar annotation might be all needed for this java application to behave as a companion sidecar to a 3rd party service (jvm or not) running in the same runtime unit (eg host, vm, docker container) whose metadata will be registered with a eureka server.

in the case of non-jvm in-house services, all is needed from them is to provide a spring boot health-like endpoint returning something like:

{
  "status":"up"
}

in the case of external services such as postgres, elastic search, kafka, etc., which likely won't provide a health check as previously described, the sideacar app itself could be used to implement such requirement:

sidecarhealthindicator.java :

package com.asimio.cloud.sidecar.healthcheck;
...
public interface sidecarhealthindicator extends healthindicator {

}

appconfig.java:

package com.asimio.cloud.sidecar.config;
...
@configuration
public class appconfig {

  @conditionalonproperty(name = "sidecar.postgres.enabled", havingvalue = "true", matchifmissing = false)
  @bean
  public sidecarhealthindicator postgreshealthcheck() {
    return new postgreshealthcheck();
  }
}

assuming a sidecar instance is going to serve as a companion to only one service, it translates to a single sidecarhealthindicator bean needed per sidecar application, a postgreshealthcheck instance in this demo.

let's back up a little bit, how can it be verified if a postgres db is accepting connections? it turns out pg_isready , a postgres command accomplishes it:

pg_isready -u postgres -h localhost -p 5432
localhost:5432 - rejecting connections

and this is what postgreshealthcheck.java does:

package com.asimio.cloud.sidecar.healthcheck.postgres;
...
public class postgreshealthcheck implements sidecarhealthindicator {
...
  // pg_isready u <user> -h localhost -p <sidecarport>
  private static final string command_pattern = "pg_isready -u %s -h localhost -p %s";

  @value("${sidecar.port}")
  private int sidecarport;

  @override
  public health health() {
    health.builder result = null;
    try {
      string output = this.runcommand();
      logger.info(output);
      if (output.indexof("accepting connections") != -1) {
        result = health.up();
      } else if (output.indexof("rejecting connections") != -1 || output.indexof("no response") != -1) {
        result = health.down().withdetail("reason", output);
      }
    } catch (ioexception e) {
      logger.warn("failed to execute command.", e);
      result = health.down().withexception(e);
    }
    return result.build();
  }
...
}

the health() method will return:

{
  "status":"up"
}

or

{
  "status":"down"
  ...
}

depending on the output of the os command .

the last piece of code needed in this case, where the service desired to register with a eureka server is outside of our control, is to expose its health information via an endpoint the eureka server can send requests to.

localstatusdelegatorcontroller.java :

package com.asimio.cloud.sidecar.web;
...
@restcontroller
public class localstatusdelegatorcontroller {

  @autowired
  private sidecarhealthindicator healthindicator;

  @requestmapping("/delegating-status")
  public health sidecarhealthstatus() {
    return this.healthindicator.health();
  }
}

the sidecar application exposes the endpoint /delegating-status which uses a health check to run a specific os command to verify if the 3rd party service is usable.

let's look at the sidecar configuration files:

# generic app name, should be passed from command line (eg java -jar ... --sidecar.appname=postgres-db_dvdrental ...)
spring:
  application:
    name: ${sidecar.appname:generic sidecar}

the sidecar.appname property value is what's going to be used for registration and discovery purposes.

application.yml:

...
sidecar:
  hostname: localhost
  # port should be passed from command line (eg java -jar ... --sidecar.port=5432 ...)
  port: 5432
  # health-uri the service uri returning health data in the form of { "status": "up" } or
  # http://localhost:${sidecar.port}/${health-uri:health.json} if the service provides such endpoint.
  health-uri: http://localhost:${server.port}/delegating-status
  # sidecar controller
  home-page-uri: http://${sidecar.hostname}:${server.port}/
  postgres:
    enabled: true
...
eureka:
  client:
    registerwitheureka: true
    fetchregistry: false
    serviceurl:
      defaultzone: http://localhost:8000/eureka/
  instance:
    appname: ${spring.application.name}
    hostname: ${sidecar.hostname}
    statuspageurlpath: ${management.context-path}/info
    healthcheckurlpath: ${sidecar.health-uri}
    preferipaddress: true
    metadatamap:
      instanceid: ${sidecar.appname}:${sidecar.port}
...

notice the sidecar.hostname property (used in eureka client configuration) is hardcoded to localhost because the sidecar companion application is supposed to run in the same host / vm / docker container / ... as the 3rd party service intended to be discovered.

other relevant settings are the sidecar.port property set to the port the 3rd party service listens on. the sidecar.health-uri property pointing to the /delegating-status endpoint, used by the eureka server to get information about the availability of the service. and sidecar.postgres.enabled , which causes the application to act as a sidecar for postgres.

it still needs to be built so that the resulting artifact could be used in a docker image along with postgres .

mvn clean package

which should create the application target/sidecar.jar .

4. set up postgres dvd rental database and sidecar application in a docker image

asimio/db_dvdrental docker image, used in integration testing using spring boot, postgres and docker would be the starting point to bundle a postgres db with the sidecar companion application .

cp target/sidecar.jar /<path to>/postgres/db_dvdrental-sidecar/scripts/

dockerfile:

from asimio/db_dvdrental:latest
maintainer orlando l otero ootero@asimio.net, https://bitbucket.org/asimio/postgres
# manually build using command: docker build -t asimio/db_dvdrental-sidecar:latest .

# install jdk
run \
  mkdir -p /usr/lib && \
  wget --header "cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/jdk-8u162-linux-x64.tar.gz  && \
  tar -zxf jdk-8u162-linux-x64.tar.gz -c /usr/lib && \
  ln -s /usr/lib/jdk1.8.0_162 /usr/lib/jdk && \
  chown -r root:root /usr/lib/jdk && \
  rm jdk-8u162-linux-x64.tar.gz  

env java_home="/usr/lib/jdk"
env path="$java_home/bin:$path"
env java_tool_options="-xms256m -xmx256m -djava.awt.headless=true -djava.security.egd=file:/dev/./urandom"

copy scripts/sidecar.jar /opt/asimio-cloud/sidecar.jar
add scripts/sidecar.sh /docker-entrypoint-initdb.d/
run chmod 755 /docker-entrypoint-initdb.d/sidecar.sh

this dockerfile bundles a postgres db, the sidecar application and a shell file to start the sidecar app in the background when a docker container is started.

sidecar.sh:

#!/bin/bash

echo "starting sidecar application"
java -jar /opt/asimio-cloud/sidecar.jar &

the image can be built running:

cd /<path to>/postgres/db_dvdrental-sidecar
docker build -t asimio/db_dvdrental-sidecar:latest .
...
successfully built 3955eae0bf98
successfully tagged asimio/db_dvdrental-sidecar:latest

note: running a docker container using this image causes the container to start two processes, which is not a suggested practice when using docker. an alternative would be to start a docker container for the sidecar application and another container for the main application, this setup requires the sidecar to know the host/ip of the main application, as well as some code changes for the sidecar to report the correct hostname to the eureka registry. still, i decided to run the two processes in the same container in this tutorial since this would be a very close approach to take when running them on bare metal or vms.

5. starting the eureka server

this demo was run using docker where communication between containers is needed, so i'll first create a docker network for them to run on:

docker network create -d bridge --subnet 172.25.0.0/16 sidecarpostgresdemo_default
4c68647c5a20c8b010f33d2fc26d3a4e751cfb4ec7571984cdcec4002313c3a6

docker run -idt -p 8000:8000 --network=sidecarpostgresdemo_default -e spring.profiles.active=standalone -e server.port=8000 -e hostname=$hostname asimio/discovery-server:1.0.73
8726b16b6abeca71dca7d882b5c30edad3c6c450959ac70b9de396bd444d8490

docker logs 87
...
2018-02-13 04:10:21.156  info 1 --- [      thread-10] e.s.eurekaserverinitializerconfiguration : started eureka server
2018-02-13 04:10:21.201  info 1 --- [           main] b.c.e.u.undertowembeddedservletcontainer : undertow started on port(s) 8000 (http)
2018-02-13 04:10:21.202  info 1 --- [           main] c.n.e.eurekadiscoveryclientconfiguration : updating port to 8000
2018-02-13 04:10:21.207  info 1 --- [           main] c.a.c.eureka.eurekaserverapplication     : started eurekaserverapplication in 6.493 seconds (jvm running for 7.121)

curl http://localhost:8000/eureka/apps
<applications>
  <versions__delta>1</versions__delta>
  <apps__hashcode></apps__hashcode>
</applications>

logs and a request to eureka server confirm it started successfully and no application has been registered yet.

6. starting the postgres dvd rental database and registering it with eureka via the sidecar app

similarly to starting the eureka server , a container including postgres and sidecar apps is started specifying the same network for containers to reach each other:

docker run -d -p 5432:5432 -p 8080:8080 --network=sidecarpostgresdemo_default -e db_name=db_dvdrental -e db_user=user_dvdrental -e db_passwd=changeit -e sidecar.port=5432 -e sidecar.appname=postgres-db_dvdrental -e eureka.client.serviceurl.defaultzone=http://172.25.0.2:8000/eureka/ asimio/db_dvdrental-sidecar:latest
252c8fc4703dde2208344e761eaa69ef6ad01b8cf5481fcb4210017848bb41bf

docker logs 25
...
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/db-init.sh
verifying db db_dvdrental presence ...
db_dvdrental db does not exist, creating it ...
verifying role user_dvdrental presence ...
user_dvdrental role does not exist, creating it ...
create role
user_dvdrental role successfully created
create database
grant
db_dvdrental db successfully created

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/db-restore.sh
importing data into db db_dvdrental
db_dvdrental db restored from backup
granting permissions in db 'db_dvdrental' to role 'user_dvdrental'.
grant
grant
permissions granted

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/sidecar.sh
starting sidecar application: fixme {sidecar.appname}
...
2018-02-13 04:40:06.683  info 136 --- [nforeplicator-0] com.netflix.discovery.discoveryclient    : discoveryclient_postgres-db_dvdrental/252c8fc4703d:postgres-db_dvdrental:8080: registering service...
...
2018-02-13 04:40:06.858  info 136 --- [           main] s.b.c.e.t.tomcatembeddedservletcontainer : tomcat started on port(s): 8080 (http)
2018-02-13 04:40:06.861  info 136 --- [           main] .s.c.n.e.s.eurekaautoserviceregistration : updating port to 8080
2018-02-13 04:40:06.867  info 136 --- [           main] c.a.cloud.sidecar.sidecarapplication     : started sidecarapplication in 7.395 seconds (jvm running for 7.924)
2018-02-13 04:40:06.982  info 136 --- [nforeplicator-0] com.netflix.discovery.discoveryclient    : discoveryclient_postgres-db_dvdrental/252c8fc4703d:postgres-db_dvdrental:8080 - registration status: 204
2018-02-13 04:40:07.087  info 136 --- [nio-8080-exec-2] o.a.c.c.c.[tomcat].[localhost].[/]       : initializing spring frameworkservlet 'dispatcherservlet'
2018-02-13 04:40:07.087  info 136 --- [nio-8080-exec-2] o.s.web.servlet.dispatcherservlet        : frameworkservlet 'dispatcherservlet': initialization started
2018-02-13 04:40:07.108  info 136 --- [nio-8080-exec-2] o.s.web.servlet.dispatcherservlet        : frameworkservlet 'dispatcherservlet': initialization completed in 20 ms
2018-02-13 04:40:07.126  warn 136 --- [nio-8080-exec-2] o.s.c.n.zuul.web.zuulhandlermapping      : no routes found from routelocator
2018-02-13 04:40:07.204  info 136 --- [nio-8080-exec-2] c.a.c.s.h.postgres.postgreshealthcheck   : localhost:5432 - accepting connections

see how a postgres db is setup first, then the sidecar companion app successfully started and ... postgreshealthcheck : localhost:5432 - accepting connections log indicates the sidecar is able to connect to the postgres server.

sending a request to the eureka server now results in a postgres-db_dvdrental service metadata stored in the registry, with [host= 172.25.0.3 , port= 5432 ].

curl http://localhost:8000/eureka/apps
<applications>
  <versions__delta>1</versions__delta>
  <apps__hashcode>up_1_</apps__hashcode>
  <application>
    <name>postgres-db_dvdrental</name>
    <instance>
      <instanceid>30117a3b2bbb:postgres-db_dvdrental:8080</instanceid>
      <hostname>172.25.0.3</hostname>
      <app>postgres-db_dvdrental</app>
      <ipaddr>172.25.0.3</ipaddr>
      <status>up</status>
      <overriddenstatus>unknown</overriddenstatus>
      <port enabled="true">5432</port>
      <secureport enabled="false">443</secureport>
      <countryid>1</countryid>
      <datacenterinfo class="com.netflix.appinfo.instanceinfo$defaultdatacenterinfo">
        <name>myown</name>
      </datacenterinfo>
      <leaseinfo>
        <renewalintervalinsecs>30</renewalintervalinsecs>
        <durationinsecs>90</durationinsecs>
        <registrationtimestamp>1518584036042</registrationtimestamp>
        <lastrenewaltimestamp>1518584036042</lastrenewaltimestamp>
        <evictiontimestamp>0</evictiontimestamp>
        <serviceuptimestamp>1518584036058</serviceuptimestamp>
      </leaseinfo>
      <metadata>
        <instanceid>postgres-db_dvdrental:5432</instanceid>
      </metadata>
      <homepageurl>http://localhost:5432/</homepageurl>
      <statuspageurl>http://localhost:8080/info</statuspageurl>
      <healthcheckurl>http://localhost:8080/health</healthcheckurl>
      <vipaddress>postgres-db_dvdrental</vipaddress>
      <securevipaddress>postgres-db_dvdrental</securevipaddress>
      <iscoordinatingdiscoveryserver>false</iscoordinatingdiscoveryserver>
      <lastupdatedtimestamp>1518584036058</lastupdatedtimestamp>
      <lastdirtytimestamp>1518584036033</lastdirtytimestamp>
      <actiontype>added</actiontype>
    </instance>
  </application>
</applications>

7. the postgres client demo application

setting up a demo client application could be done following a similar approach as when creating the sidecar application .

pom.xml:

...
<dependency>
  <groupid>org.springframework.cloud</groupid>
  <artifactid>spring-cloud-starter-eureka</artifactid>
</dependency>
<dependency>
  <groupid>org.springframework.boot</groupid>
  <artifactid>spring-boot-starter-web</artifactid>
</dependency>
<dependency>
  <groupid>org.springframework.boot</groupid>
  <artifactid>spring-boot-starter-data-jpa</artifactid>
</dependency>
<dependency>
  <groupid>org.postgresql</groupid>
  <artifactid>postgresql</artifactid>
</dependency>
...

these are the main dependencies needed for the demo postgres client application to locate and connect to a postgres db.

sidecarpostgresdemoapplication.java :

package com.asimio.demo;
...
@springbootapplication
@enableeurekaclient
public class sidecarpostgresdemoapplication {

  public static void main(string[] args) {
    springapplication.run(sidecarpostgresdemoapplication.class, args);
  }
}

@enableeurekaclient annotation along with configuration properties allow this application to register and/or discover metadata from a eureka server.

appconfig.java:

package com.asimio.demo.config;
...
@configuration
@enableconfigurationproperties({ datasourceproperties.class })
public class appconfig {
  ...
  @autowired
  private datasourceproperties dsproperties;

  @autowired
  private discoveryclient discoveryclient;

  @value("${sidecar.appname:postgres-db_dvdrental}")
  private string dbservicename;

  @bean
  public datasource datasource() {
    ...
    serviceinstance instance = this.discoveryclient.getinstances(this.dbservicename).iterator().next();
    ...
    return this.createdatasource(instance.gethost(), instance.getport());
  }

  private datasource createdatasource(string host, int port) {
    string jdbcurl = string.format(this.dsproperties.geturl(), host, port);
    ...
    datasourcebuilder factory = datasourcebuilder
        .create()
        .url(jdbcurl)
        .username(this.dsproperties.getusername())
        .password(this.dsproperties.getpassword())
        .driverclassname(this.dsproperties.getdriverclassname());
    return factory.build();
  }
}

a datasource bean is instantiated instead of relying in datasourceautoconfiguration because the hostname and port the postgres server listens on is unknown until eureka responds back with metadata.

application.yml:

spring:
  autoconfigure:
    exclude:
      - org.springframework.boot.autoconfigure.jdbc.datasourceautoconfiguration
  datasource:
    name: db_dvdrental
    # placeholder to be replaced at runtime with metadata retrieved from registration server
    url: jdbc:postgresql://%s:%s/db_dvdrental
    username: user_dvdrental
    password: changeit
    driverclassname: org.postgresql.driver

  jpa:
    database: postgresql
    database-platform: org.hibernate.dialect.postgresqldialect
    generate-ddl: false
    hibernate:
      ddl-auto: none
...
sidecar.appname: postgres-db_dvdrental

eureka:
  client:
    registerwitheureka: false
    fetchregistry: true
    serviceurl:
      defaultzone: http://localhost:8000/eureka/
...

notice the jdbc url includes placeholders to be replaced with values coming from eureka.

mvn clean package -dskiptests
...
mvn docker:build
...

i'll skip running the integration tests for now because this demo application needs a eureka and postgres servers to connect to while loading the spring context. in integration testing using spring boot, postgres and docker i covered how to start dependent services before running each test.

8. starting the postgres client demo

running this demo in a docker container requires the usage of the same network the other two containers use:

docker run -idt -p 8090:8090 --network=sidecarpostgresdemo_default -e server.port=8090 -e eureka.client.serviceurl.defaultzone=http://172.25.0.2:8000/eureka/ asimio/sidecar-postgres-demo:latest
671a60617aab2134e2a037519b56e349beb01b5c88787d6280d043e21e85ef5b

docker logs 67
...
2018-02-13 05:03:48.248  info 1 --- [           main] s.b.c.e.t.tomcatembeddedservletcontainer : tomcat started on port(s): 8090 (http)
2018-02-13 05:03:48.249  info 1 --- [           main] .s.c.n.e.s.eurekaautoserviceregistration : updating port to 8090
2018-02-13 05:03:48.255  info 1 --- [           main] c.a.demo.sidecarpostgresdemoapplication  : started sidecarpostgresdemoapplication in 9.586 seconds (jvm running for 10.059)

now that it has started, let's send a couple of requests to an endpoint which should successfully execute a db query:

curl http://localhost:8090/actors/1
[actor: penelope guiness]

curl http://localhost:8090/actors/2
[actor: nick wahlberg]

a relevant note before finishing showing how to implement the sidecar pattern using postgres, spring cloud netflix, and docker.

you probably noticed i have started the eureka server docker container first, then a postgres container bundled with a sidecar java application and lastly the demo restful service container which connects to a postgres db once it finds its metadata from eureka. i actually included a docker compose file in an attempt to simplify and automate this process but not without its challenges.

connection-related problems arose when starting the demo api application but the eureka server hasn't started yet or when the postgres host metadata is not still available in the eureka server while instantiating the datasource bean .

i believe it would be a good practice for microservices to recover themselves, to self-heal from a situation like this, where services startup order is ideal but not required, where an application would keep trying to connect to dependent services for a given time and/or a number of attempts. would this be a concern of the application or a concern of some kind of platform orchestrator? stay tuned , i might follow up with an implementation of this approach in another post.

9. source code

accompanying source code for this blog post can be found at:

• eureka server

• sidecar application

• db_dvdrental-sidecar postgres docker image

• sidecar postgres demo

10. references

• http://cloud.spring.io/spring-cloud-netflix/single/spring-cloud-netflix.html#_polyglot_support_with_sidecar

• https://github.com/spring-cloud/spring-cloud-netflix/tree/master/spring-cloud-netflix-sidecar

• https://docs.docker.com/engine/userguide/networking/work-with-networks/#basic-container-networking-example

• https://docs.docker.com/compose/networking/