introduction

in this article i’m going to explain how you can setup a real time sensors dashboard using google app engine and a raspberry pi zero . here is a live demo and the video below shows the end result. if this sounds interesting, then read on for the details:

google app engine is a platform as a service (paas) that lets you deploy and run your applications on the google infrastructure without having to worry about setting up your own hardware, operating system or server. we will also use google charts — free, powerful and simple charting tools — to plot the sensors data into line charts. we have also used initializr — a html5 templates generator — to generate a template for the dashboard which includes bootstrap , jquery and other useful frontend resources.

for the hardware we will use the glorious raspberry pi zero, a fully fledged computer smaller than a credit card. the pi zero features a 1ghz single-core cpu, 512mb ram, mini hdmi and usb and a 40 pin gpio header. we will connect a few sensors to the gpio pins and send their data over to google app engine. when user views the dashboard both the values and charts on the dashboard update in real time whenever new data arrives from the sensors. the source code for both the app engine dashboard and the pi zero app is in github with instructions on how to build and deploy each project.

architecture overview

in this article we use java as the programming language, java is supported by both google app engine and the pi zero through the pi4j library. however, if you prefer you can easily rework the code into python, which is also supported by both the pi zero and google app engine. the new versions of the pi zero operating system, raspbian , comes with oracle java 8 pre-installed. ultimately we deploy and run an executable jar on the pi zero, this jar reads the input from the sensors and updates google app engine. we use apache maven to compile and build the code on the pi zero, although this is not necessary as you can build the code on your laptop and copy the jar over to the pi zero.

the full architecture is shown below:

on the google app engine side we use cloud endpoint , a very powerful service that we can use to create a backend api and client libraries for mobile and web by using annotations. since the generated android client is java based we can use it with the pi zero application. the api that the pi zero calls is authenticated using google oauth 2.0 for installed applications . we use an authenticated api to ensure that only our authorised pi zero is actually able to send updates to the server. we also use the channel api , one of the google app engine services that enables us to establish a persistent connection between the browser and the server so we can push real time messages to the browser. this is obviously better than continuously polling the server for updates.

the hardware

the hardware we are building in this article is basic and is inspired from other online articles. instead of repeating the setup here i will reference these articles where appropriate. here is the breadboard view of the completed circuit generated using fritzing . if you install fritzing you can download this schematic here :

the purpose of this hardware is to provide the readings of three sensors, voltage from a solar cell, temperature from an analogue temperature sensor and illuminance (lux) using a photocell. because the output of the temperature sensor and photocell is analogue and the pi zero only accepts digital input, we use an analogue to digital converter (adc).

here are the components used:

• resisters: 3 x 220ω, 1 x 10kω, 1 x 100kω and 1 x 150kω
• general purpose diode (for the solar cell)
• three leds (red, yellow and green)
• generic photocell
• analog temperature sensor: adafruit tmp36
• mini solar cell 6.5 x 6.5cm 5.5v 90ma 0.6w
• analogue to digital converter: adafruit mcp3008 8-channel 10-bit adc with spi interface
• raspberry pi 40 pins coupler and cable
• breadboard
• raspberry pi zero with official wifi dongle and adapter kit .

if you use the pi zero then you need to solder pins on the unpopulated gpio headers . we use the 3.3v pin 1 on the pi zero to power our various components. we use a red led connected to the pi’s 3.3v pin as an indication of power, a green led as a load for the solar cell and a yellow led to indicate when the pi is saving data to the cloud. the yellow led is connected to pin 18 on the pi zero. all leds are connected to 220ω resistors.

connecting the sensors

since all the three components, temperature sensor, photocell and solar cell output analogue voltage between 0 and 3.3v, we use the mcp3008 adc to convert these analogue voltage values into digital values between 0 and 1023. this is because the mcp3008 is a 10 bit adc and the largest decimal number you can represent with 10 bits is 2^10 - 1 = 1023. the mcp3008 adc has 8 input channels, which means we can connect up to 8 analogue inputs to it. the mcp3008 uses the serial peripheral interface (spi) for communication, which is already supported by the pi zero. bear in mind to program the mcp3008 adc we need to understand the structure of the data that should be sent to it, these are explained on the datasheet on page 21. the pi zero code is full of comments that explains why particular values are being sent to the adc.

we connect the photocell to channel 0, the temperature sensor to channel 1 and the solar cell to channel 2, this leaves the other 5 channels free. the photocell and temperature sensor connections used here are the same as in this raspberrypi-spy article , so we won’t repeat this setup here. be sure to follow this article on the same website to enable the spi interfaces on your pi.

temperature

the temperature formula we use for the tmp36 is based on the one on this article on the adafruit website:

millivolts = read_adc0 * ( 3300.0 / 1023.0 )
temp_c = ((millivolts - 100.0) / 10.0) - 40.0

illuminance

again, to measure the illuminance we use a lookup table based on these values on the adafruit website, bear in mind these are just an approximation because we are using a generic photocell without the datasheet. if you want more accurate lux readings then it’s better to use a digital lux sensor.

voltage

for the solar cell a diode is connected to its positive terminal as described here . additionally, because the cell can produce a maximum of 5.5v we use a voltage divider to ensure the adc receives only 3.3v when the cell output is at its maximum  (5.5v). the circuit on the left show the solar cell with resister r2 = 220ω and green led3 as load. we then use r6 and r7 as voltage divider so that we can calculate the cell output voltage from the voltage read by the adc using this formula:

this circuit for measuring the solar cell voltage is inspired by this article , which includes the wiring and the formula for measuring the current as well. this article on raspi.tv for measuring the voltage output of a 2 cell lithium polymer (lipo) battery is also useful. here is the fully assembled breadboard and pi zero:

the dashboard app

the dashboard app is a simple java web application that is deployed to google app engine. the source code is located in github . here we will provide a brief review of each of the key files. the instructions on how to deploy the application are on github, simply follow them to get your own dashboard up and running on google app engine.

the java classes are organised in three packages, the endpoint package contains the sensor data api class. the entities package contains the objects that we persist to the datastore (the app engine nosql database). the servlets package contains the java web application servlets. the most important class is the sensordataendpoint.java that implements the sensor data api.

the sensor data api

the sensor data api is coded using cloud endpoints and implemented in sensordataendpoint.java shown below. as you can see we have created a fully fledged api by using annotations such as @api on line 23 and @apimethod on line 37. google app engine will expose the create method annotated with @apimethod as a rest/rpc api that uses json over http. the client libraries for android and ios can also be created by the user.

package uk.co.inetria.pi.endpoint;

import java.util.list;
import java.util.logging.level;
import java.util.logging.logger;

import com.google.api.server.spi.config.api;
import com.google.api.server.spi.config.apimethod;
import com.google.appengine.api.channel.channelmessage;
import com.google.appengine.api.channel.channelservice;
import com.google.appengine.api.channel.channelservicefactory;
import com.google.appengine.api.users.user;
import com.google.gson.gson;

import uk.co.inetria.pi.entities.client;
import uk.co.inetria.pi.entities.sensordata;


/**
  * add your first api methods in this class, or you may create another class. in that case, please
  * update your web.xml accordingly.
 **/
@api(
  name = "sensor",
  version = "v1",
  scopes = {constants.email_scope},
  clientids = {constants.web_client_id, constants.android_client_id, constants.ios_client_id},
  audiences = {constants.android_audience}
)
public class sensordataendpoint {

  private static final logger log = logger.getlogger(sensordataendpoint.class.getname());

  private static final gson gson = new gson();


  @apimethod(name = "data.create", httpmethod = "post")
  public sensordata create(sensordata data, user user) {

    // check the user is authenticated and authorised
    if(user == null) {
      log.warning("user is not authenticated");
      throw new runtimeexception("authentication required!");

    } else if(!constants.email_address.equals(user.getemail())) {

      log.warning("user is not authorised, email: " + user.getemail());
      throw new runtimeexception("not authorised!");
    }

    data.save();

    try {
      // notify the client channels
      channelservice channelservice = channelservicefactory.getchannelservice();

      list<client> clients = client.findall();
      string json = gson.tojson(data);

      for(client client: clients) {
          channelservice.sendmessage(new channelmessage(client.getid(), json));
      }

    } catch(exception e) {
        log.log(level.severe, "failed to notify connected clients", e);
    }

    return data;
  }

}

when the pi zero calls this create method, google app engine will populate and inject a user object if the cloud endpoint request was authenticated using a valid google account. if not, we throw some exceptions, otherwise we go ahead and save the sensordata object. note that app engine is doing the marshalling and unmarshalling of the sensordata object from json to java and vice versa. next step on line 55 we use the app engine channel api to notify any connected clients (browsers) that we have new data. we retrieve all active clients from the datastore and update each of their channels.

http requests handlers

when a user loads the dashboard into their browser the javascript makes an ajax call to /getdata, this is mapped to the startservlet.java .

package uk.co.inetria.pi.servlets;

import java.io.ioexception;
import java.util.calendar;
import java.util.date;
import java.util.gregoriancalendar;
import java.util.hashmap;
import java.util.list;
import java.util.map;
import java.util.logging.logger;

import javax.servlet.servletexception;
import javax.servlet.http.httpservlet;
import javax.servlet.http.httpservletrequest;
import javax.servlet.http.httpservletresponse;
import javax.servlet.http.httpsession;

import com.google.appengine.api.channel.channelservice;
import com.google.appengine.api.channel.channelservicefactory;
import com.google.gson.gson;

import uk.co.inetria.pi.entities.client;
import uk.co.inetria.pi.entities.sensordata;


/**
 * servlet implementation class startservlet
 */
public class startservlet extends httpservlet {

  private static final long serialversionuid = 1l;

  private static final logger log = logger.getlogger(startservlet.class.getname());

  private static final string content_type_json = "application/x-json";

  private static final gson gson = new gson();

  /**
   * @see httpservlet#doget(httpservletrequest request, httpservletresponse response)
   */
  protected void doget(httpservletrequest request, httpservletresponse response) throws servletexception, ioexception {

    // get the session 
    httpsession session = request.getsession();
    string clientid = session.getid();

    log.info("sending data to client " + clientid);

    client client = client.findbyid(clientid);

    string token = null;

    if(client == null) {

        channelservice channelservice = channelservicefactory.getchannelservice();
        log.info("creating channel for clientid " + clientid);
        // open the channel
        token = channelservice.createchannel(clientid);

        client = new client(clientid, token);
        client.save();
    }

 // use the session id and the channel id for this client
    map<string, object> data = new hashmap<>();
    token = client.gettoken();
    data.put("token", token);

    //createtestdata();
    int minutes = 1;
    string timerange = request.getparameter("timerange");

    try {
        minutes = integer.parseint(timerange);

    } catch(numberformatexception e) {
        log.warning("failed to parse number " + timerange);
    }

    calendar calendar = new gregoriancalendar();
    calendar.add(calendar.minute, -minutes);
    date date = calendar.gettime();

    list<sensordata> sensordatalist = sensordata.findbydatetime(date);

    data.put("data", sensordatalist);

    response.setcontenttype(content_type_json);
    response.getwriter().print(gson.tojson(data));

  }

}

this servlet will use the http session id to determine if the user has a client entity saved in the datastore, which mean they already have an open channel. if not, then a new client entity and a channel are created, the http session id is used to generate a channel token , which is sent to the client. the javascript uses this token to establish a channel connection and registers a handler which is invoked when messages are received. the servlet then retrieves the sensor data for the last x minutes and sends a response back to the client as json.

the channelhandler.java class handles channels connection and disconnection , whenever a new client connects to or disconnects from a channel this handler is invoked. when a client disconnects we use this handler to remove their corresponding client entity from the datastore, so we don’t have to worry about updating their channel anymore.

the frontend code

the bulk of the frontend code is in main.js , in here we make an ajax call to get the channel token and a list of sensor data for the last x minutes. once we get the data from app engine we reformat it so that it’s suitable for the google line chart . we then use the channel token to establish a channel connection and register an event handler for when channel messages are received.

the user interface offers the user three charts with the ability to retrieve data for the last minute, hour, day and 30 days. the user can also choose to auto refresh the charts when new sensor data is received through the channel api. we use the jquery animate number plugin to add a nice effect when new sensor values are received.

the raspberry pi app

the raspberry pi app is a java jar with an executable main class. the source code for the app is in github . the app consists of a bunch of java files, the files highlighted in red are the ones we’ve created. the other classes are automatically generated by the app engine cloud endpoints utility and we just copied and pasted them in the app. these classes represent the cloud endpoints client that the pi is going to call when it’s sending sensor data to app engine.

the cmdlineauthenticationprovider.java file provides the implementation for an oauth 2.0 installed client . when the application is started from the command-line for the first time it asks the user to type in a url into their browser:

$ java -jar raspberrypiapp.jar 
please open the following address in your browser:

https://accounts.google.com/o/oauth2/auth?client_id=111109056482-6jn34uijapemnbfk4k7ukcv7eclse1ln.apps.googleusercontent.com&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code≻ope=https://www.googleapis.com/auth/userinfo.email
attempting to open that address in the default browser now...
please enter code: afdsadsfasdfawikehuzimostk9hz96vkfmbza0

once the url is opened on the browser, it will forward to the google accounts ‘request for permissions’ page. we need to be signed into a google account and accept the request. a short code will be generated which we then need to copy and paste into our pi app as shown in ‘ please enter code: ‘ snippet above. once that is done, the application will now exchange this code for an oauth 2.0 token that can be used to make requests on our behalf.

the raspberrypiapp.java is where everything else apart from authentication takes place. here the values of the sensors are read and sent over to google app engine using cloud endpoints. the two noteworthy methods are shown below. readchannel , sends 3 bytes to the mcp3008 chip using the spi interface and gets back 3 bytes, from which we extract 10 bits. the logic is explained in the comments below, uses java bitwise operators and is based on the python code in this article . the other sendsensordata method uses the cloud endpoints clients to send the sensor data over to google app engine.

/**
 * function to read spi data from mcp3008 chip channel must be an integer
 * 0-7 based on this python code (see
 * http://www.raspberrypi-spy.co.uk/2013/10/analogue-sensors-on-the-
 * raspberry-pi-using-an-mcp3008/) adc = spi.xfer2([1,(8+channel)<<4,0])
 * data = ((adc[1]&3) << 8) + adc[2]
 * 
 * we also make heavy use of bitwise operator, see here for more info
 * https://docs.oracle.com/javase/tutorial/java/nutsandbolts/op3.html
 */
private static int readchannel(int channel) {

  byte packet[] = new byte[3]; // we send 3 bytes to communicate with the
  // mcp3008 chip
  packet[0] = 0b00000001; // start bit
  packet[1] = (byte) ((8 + channel) << 4); // data,explained below
  // 8 = 0b1000, add channel, shift left 4 places. chip expects 0b10010000
  // to read channel 1
  packet[2] = 0b00000000; // chip doesn't care!

  spi.wiringpispidatarw(0, packet, 3);

  // extract a 10 bits number from the 3 bytes returned (explained below)
  int data = ((packet[1] & 0b11) << 8) | (packet[2] & 0xff); // & 0xff to loose the sign, java bytes & ints are signed!
  // chip returns 3 bytes at indexes, 0, 1 & 2
  // ?_?_?_?_?_?_?_?, ?_?_?_?_?_0_b9_b8, b7_b6_b5_b4_b3_b2_b1_b0
  // ?_?_?_?_?_0_b9_b8 & 0b11 = b9_b8
  // 0_0_0_0_0_0_b9_b8 << 8 = b9_b8_0_0_0_0_0_0_0_0
  // b9_b8_0_0_0_0_0_0_0_0 + packet[2] = b9_b8_0_0_0_0_0_0_0_0 +
  // b7_b6_b5_b4_b3_b2_b1_b0 = b9_b8_b7_b6_b5_b4_b3_b2_b1_b0

  return data;
}

/**
 * 
 * @param value
 * @param channel
 * @param sensor
 * @throws ioexception
 */
private static void sendsensordata(double value, string channel, sensor sensor, gpiopindigitaloutput led1)
throws ioexception {

  // continuously blink the led every 1/2 second for 3 seconds
  led1.blink(500, 3000);
  // test creating some sample data
  sensordata data = new sensordata();
  data.setchannel(channel);
  data.setdatetime(new datetime(new date()));
  data.setvalue(value);

  sensor.data().create(data).execute();
}

summary

in this article we have presented a full working solution for implementing a real time sensors dashboard ( https://raspberrypi-dash.appspot.com/ ) on google app engine platform as a service. we have used the following google app engine technologies:

• cloud endpoints
• channel api
• datastore

we have also used google oauth 2.0 to authenticate updates from our raspberry pi zero. on the frontend side we have used bootstrap , google charts and jquery . for the raspberry pi, we have tested this app on pi zero and used the pi4j library to be able to interact with the gpio pins using java.

where to next

you can deploy and run both applications as detailed in github, you can also modify both the dashboard and pi code to suit your need:

• app engine dashboard: https://github.com/omerio/raspberrypi-appengine-portal
• raspberry pi app: https://github.com/omerio/raspberrypi-app

if you are interested in implementing internet of things (iot) solutions on the google cloud platform, or wondering about the best architecture to follow then make sure to checkout the iot documentations .