Developing IoT Mashups with Docker, MQTT, Node-RED, InfluxDB, Grafana

Page for the tutorial " Developping IoT Mashups with Docker, MQTT, Node-RED, InfluxDB, Grafana " at Eclipse IoT Days Grenoble 2016

COMING SOON

=Install Docker=

=Install Node.js and NPM= On Raspian sudo apt-get install nodejs npm

=Install Node-RED=

On Raspian Jessie sudo apt-get update sudo apt-get install nodered Or sudo npm install -g --unsafe-perm node-red

The default directory is : ~/.node-red

Browse http://127.0.0.1:1880/

Now, you can write your own Node-RED flows and functions.

Install the admin cmdline tool

sudo npm install -g node-red-admin node-red-admin help

node-red-admin list
 * 1) installed modules

node-red-admin search serial node-red-admin search rfxcom node-red-admin search zwave node-red-admin search ble node-red-admin search bluetooth node-red-admin search sensortag
 * 1) search modules

node-red-admin search beaglebone node-red-admin search gpio

node-red-admin search mongodb node-red-admin search redis node-red-admin search cassandra node-red-admin search elastic node-red-admin search influxdb node-red-admin search kafka

Remark: node-red-admin install does not work ! Use

cd ~/.node-red sudo npm install -g node-red-node-redis sudo npm install -g node-red-contrib-kafka-consumer sudo npm install -g node-red-node-mongodb

Restart Node-Red

node-red-admin list

=Install extra nodes for Node-RED= Extra nodes are provided with the Node-RED community. There are listed here.

Serial for Geiger Counter


Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

Connect the Geiger counter to the host.

Check available serial ports (/dev/tty.usbserial* on MacOS X, ...) with.

Add a node serial "Geiger" with a Settings of 9600/8/N/1 and 'Split input into fixed lenghts of 1 chars'.

Add a node debug.

Connect "Geiger" to debug.

Deploy the flow.

The Geiger Counter sends a random sequence of 0 and 1.

Add a node function "Count Particles" with a flow-scoped variable (geiger/count): var COUNT='geiger/count';

// initialise the counter to 0 if it doesn't exist already var count = flow.get(COUNT)||0; count += 1; // store the value back flow.set(COUNT,count); // make it part of the outgoing msg object msg.count = count; msg.payload = {}; msg.payload.device = "geiger"; msg.payload.count = count;

node.log("Received particles : " + count);

return msg;

Connect node "Geiger" to node "Count Particles".

Deploy the new flow.

Edit the node "Count Particles" and add the 2 following statements in order to display the count into the node's status. ... setTimeout(function { node.status({}); }, 1000); // The shape property can be: ring or dot. // The fill property can be: red, green, yellow, blue or grey node.status({fill:"green",shape:"dot",text:"#"+count}); return msg;

Deploy the new flow.

Add a node inject "One minute timer" with a repeat interval of 1 minute.

Add a node function "Reset Particles Count" with a flow-scoped variable (geiger/count): var COUNT='geiger/count';

// initialise the counter to 0 if it doesn't exist already var count = flow.get(COUNT)||0;

msg.count = count; msg.payload = {}; msg.payload.device = "geiger"; msg.payload.ppm = count;

node.log("Reset counter at " + count);

// make it part of the outgoing msg object count = 0; // store the value back flow.set(COUNT,count);

return msg;

Connect node "One minute timer" to node "Reset Particles Count" and node "Reset Particles Count" to node debug.

Deploy the new flow.

The result is:

Install RFXCom for Oregon Sensors


Install the rfxcom node (node-red-contrib-rfxcom) cd ~/.node-red npm install -g node-red-contrib-rfxcom

Restart Node-RED with.

Check the module in the list node-red-admin list | grep rfx

Connect the RFXCom receiver to the host.

Check available serial ports (/dev/tty.usbserial* on MacOS X, ...) with.

Add a node rfxcom-sensors "RFXCom" with the correct serial port.

Add a node debug display the full message (not only msg.payload).

Connect "RFXCom" to debug.

Deploy the flow.

The flow loos like that:

Serial for Arduino
Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

TO BE CONTINUED

Serial for LoRa Libelium
Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

Serial for LoRa Nucleo
Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

TO BE CONTINUED

UDP for LoRaWAN
Add a node 'udp in' listening the port 8123.

Add a node 'function' "toString" with the following statement: msg.payload=msg.payload.toString; return msg;

Add a node 'debug'.

Connect them together.

Deploy the new flow.

Test the flow with the shell command echo -n -e '{"device"="123456","temperature"=37.2}' > /dev/udp/127.0.0.1/8123

The flow looks like that:

Add a node 'json' into the flow.

Now you can filter and send the output of a process to the flow througth an 'udp in' node.

TO BE CONTINUED

UDP for ESP8266
Add a node 'udp in' listening the port 8123.

Add a node 'function' "toString" with the following statement: msg.payload=msg.payload.toString; return msg;

Add a node 'debug'.

Connect them together.

Deploy the new flow.

Test the flow with the shell command echo -n -e "{'device'='123456','temperature'=37.2}" > /dev/udp/127.0.0.1/8123

Add a node 'json' into the flow.

ACR122U NFC Reader
Plug the ACR122U NFC reader in the host and check it with

Install LibNFC (on Debian) sudo apt-get install libnfc-dev libnfc-bin libnfc-examples

sudo nfc-list

Launch nfc-poll and put a NFC card/tag on the reader sudo nfc-poll -v

The ID of the NFC card/tag is prefixed by the string.

Execute the following script :

while true do nfc-poll | grep "UID (NFCID1):" > /tmp/nfc.log done
 * 1) !/bin/sh

Into the Node-RED dashboard, create a new flow.

Add a node tail for reading the tail of the log file containing the card/tags identifiers.

Add a node debug.

Connect the node tail to the node debug.

Deploy the new flow.

Add a node function "Build message" with the following statements: var m=msg.payload; var i=m.indexOf(":"); if(i>=0) { var tid = m.substr(i+1).replace(/\s+/g, ''); msg.payload={} msg.payload.device="acr122u"; msg.payload.nfc=tid; return msg; } return null;

Connect the node tail to the node "Build message" and the node "Build message" to the node debug.

Deploy the new flow.

Edit the node "Build message" and add the 2 following statements in order to display the last tid into the node's status during one second. ...   setTimeout(function { node.status({}); }, 1000); // The shape property can be: ring or dot. // The fill property can be: red, green, yellow, blue or grey node.status({fill:"green",shape:"dot",text:"#"+tid}); return msg; } return null;

Deploy the new flow.

=Install Mosquitto=

On MacOS X brew install mosquitto

On Debian sudo apt-get install mosquitto sudo service mosquitto status sudo apt-get install mosquitto-clients

Test local Mosquitto broker BROKER=localhost mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' & mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="123456","temperature"=37.2}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="345678","temperature"=23.2,"humidity"=50}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0002' -m '{"device"="acr122u","nfc"="04ddf0f9232580"}' sleep 1 pkill mosquitto_sub

Test public Mosquitto broker BROKER=test.mosquitto.org mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' & mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="123456","temperature"=37.2}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="345678","temperature"=23.2,"humidity"=50}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0002' -m '{"device"="acr122u","nfc"="04ddf0f9232580"}' pkill mosquitto_sub

=Using MQTT for collecting sensors data=

In the flows defined above, add a node mqtt out with a new MQTT broker localhost:1883 and a topic iotdays/sensors/gwXXXX where gwXXXX is the name of the host.

Check the publishing with BROKER=localhost mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' &

Create a new flow for subscribing messages from the Mosquitto MQTT broker.

Add a node debug for displaying the full message.

Add a node mqtt in with with a new MQTT broker localhost:1883 and a topic iotdays/sensors#

Connect the node mqtt in to the node debug.

Deploy the new flow.

=Install InfluxDB=

=Populate InfluxDB from Node-RED=

influx

Enter the following InfluxDB statements create database iotdb use iotdb show measurements show series select * from temperature select * from humidity select * from nfc select * from ppm

In the Node-RED dashboard, create a new flow to collect sensors data from the MQTT server.

Install the Node-RED InfluxDB node.

Restart Node-RED.

Add a node mqtt in "MQTT Broker" for subscribing to the MQTT server (broker is test.mosquitto.org or localhost topic is iotdays/sensors/#)

Add a node function "Tranform into time series" to format the JSON messages into time series messages.

Add a node influxdb out "IoT Database" with the InfluxDB server.

Connect the node "MQTT Broker" to node "Tranform into time series" and to the node "Tranform into time series" to node "IoT Database".

The flow looks like than:

=Install Grafana=

See Grafana

=Visualize sensors data into the Grafana dashboard=

=Install Apache Spark=

See Spark

import org.eclipse.paho.client.mqttv3._ import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence

import org.apache.spark.storage.StorageLevel import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark.streaming.mqtt._ import org.apache.spark.SparkConf

val brokerUrl = "tcp://localhost:1883" val topic = "iotdays/sensors/#" val ssc = new StreamingContext(sc, Seconds(60)) val sensordatas = MQTTUtils.createStream(ssc, brokerUrl, topic, StorageLevel.MEMORY_ONLY_SER_2)

TODO

ssc.start ssc.awaitTermination

=Process a stream of sensors data=