Ingesting IoT Sensor Data Into S3 With an RPI3

Due to the increasing amount of data produced from outside source systems, enterprises are facing difficulties in reading, collecting, and ingesting data into a desired, central database system. An edge pipeline runs on an edge device with limited resources, receives data from another pipeline or reads the data from the device, and controls the device based on the data.

StreamSets Data Collector (SDC) Edge, an ultra-lightweight agent, is used to create end-to-end data flow pipelines in StreamSets Data Collector and to run the pipelines to read and export data in and out of systems. In this blog, StreamSets Data Collector Edge is used to read data from an air pressure sensor (BMP180) from an IoT device (Raspberry Pi3). Meanwhile, StreamSets Data Collector is used to load the data into Amazon's Simple Storage Service (S3) via MQTT.

Prerequisites

• Install StreamSets
• Raspberry Pi3
• BMP180 Sensor
• Amazon S3 Storage

Use Case

• Read an air pressure sensor data with an IoT Device (Raspberry Pi3) and send data via MQTT
• Use SDC to load the data into Amazon S3 via MQTT

Synopsis:

• Connect the BMP180 temperature/pressure sensor with your Raspberry Pi3
• Create an edge sending pipeline
• Create a data collector receiving pipeline

Flow diagram:

Connecting BMP180 Temperature/Pressure Sensor With a Raspberry Pi3

I2C bus, a communication protocol, is used by Raspberry Pi3 to communicate with other embedded IoT devices such as temperature sensors, displays, accelerometers, and so on. The I2C bus has two wires called SCL and SDA. SCL is a clock line to synchronize all data transfers over the I2C bus, and the SDA is a data line. The devices are connected to the I2C bus via the SCL and SDA lines.

To enable I2C drivers on the Raspberry Pi3, perform the following:

• Run sudo raspi-config.
• Choose Interfacing Options from the menu as shown in the following image:

• Choose I2C as shown in the image below:

Note: If I2C is not available in the Interfacing Options, check Advanced Options for I2C availability.

• Click Yes to enable the I2C driver.
• Click Yes again to load the driver by default.
• Add i2c-dev to /etc/modules using the following commands:

pi@raspberrypi:~$ sudo nano /etc/modules
i2c-bcm2708
i2c-dev

• Install i2c-tools using the following command:

pi@raspberrypi:~$ sudo apt-get install python-smbus i2c-tools

• Reboot the Raspberry Pi3 by using the following command:

sudo reboot

• Ensure that the I2C modules are loaded and made active using the following command:

pi@raspberrypi:~$ lsmod | grep i2c

• Connect the Raspberry Pi3 with the BMP180 temperature/pressure sensor as shown in the diagram below:

• Ensure that the hardware and software are working fine with i2cdetect using the following command:

pi@raspberrypi:~$ sudo i2cdetect -y 1

Building Edge Sending Pipeline

To build an edge sending pipeline to read the sensor data, perform the following:

• Create an SDC Edge Sending pipeline on StreamSets Data Collector.
• Read the data directly from the device (using I2C Address) using “Sensor Reader" component.
• Set the I2C address as “0x77”.
• Use an Expression Evaluator to convert the temperature from Celsius to Fahrenheit.
• Publish data to the MQTT topic as “bmp_sensor/data”.
• Download and move the SDC Edge pipeline's executable format (Linux) to device side, where the pipeline runs on the device side (Raspberry Pi3).
• Start SDC Edge from the SDC Edge home directory on the edge device using the following command:

bin/edge –start=<pipeline_id>

For example:

bin/edge --start=sendingpipeline137e204d-1970-48a3-b449-d28e68e5220e

Building Data Collector Receiving Pipeline

To build a data collector receiving pipeline for storing the received data in Amazon S3, perform the following:

• Create a receiving pipeline on the StreamSets Data Collector.
• Use the MQTT subscriber component to consume data from the MQTT topic (bmp_sensor/data).
• Use the Amazon S3 destination component to load the data into Amazon S3.
• Run the receiving pipeline in the StreamSets Data Collector.

The real-time air pressure data collected and stored is shown in the image below.