Developing IoT Mashups with Docker, MQTT, Node-RED, InfluxDB, Grafana

Page for the tutorial " Developping IoT Mashups with Docker, MQTT, Node-RED, InfluxDB, Chronograf, Grafana " at Eclipse IoT Days Grenoble 2016

=Introduction=

The goal of this tutorial is building rapidly a minimal IoT stack from sensors to dataviz and realtime analytics.

The hardware compoments used in this stack are:
 * various (wireless) sensors,
 * USB radio modems plugged into embedded IoT gateways,
 * Raspberry Pi 1 and 2 playing the role of embedded IoT gateways,
 * Your PC/Mac playing the role of the IoT datacenter server,
 * A t2 micro AWS instance playing the role of the IoT datacenter server if a public IP address is required (case for Sigfox callback).

The software compoments used in this IoT stack are :
 * Docker,
 * Node.js and NPM,
 * Node-RED and node contributions,
 * Mosquitto
 * InfluxDB
 * MongoDB
 * Chronograf
 * Grafana (not on RPI)
 * Spark (not on RPI)

Those software compoments are preinstalled on the Raspian Jessie distribution of the Raspberry PI boards in order to speed the installation and prevent WiFi whims.

Tools are:
 * vi
 * Arduino IDE
 * Waspmote IDE
 * MBed IDE
 * Energia for TI SensorTag2

Security, Avaibility and Performance issues are not in the scope of this tutorial.

Radio communication technologies are : Wifi, BLE, LoRa, SigFox, NFC and Rfxcom 433 MHz.

Info: the username and password on the RPIs are  and  

=Install Docker=

MacOS and Debian
See Docker

Raspian
wget --no-check-certificate https://downloads.hypriot.com/docker-hypriot_1.10.2-1_armhf.deb sudo dpkg -i docker-hypriot_1.10.2-1_armhf.deb sudo service docker restart sudo docker info

=Install Node.js and NPM=

http://nodered.org/docs/hardware/raspberrypi

On Raspian Jessie : Node.JS is already installed but NPM not nodejs -v sudo apt-get update sudo apt-get install npm npm -v

On Raspian Wheezy sudo apt-get update sudo apt-get install nodejs npm nodejs -v npm -v

On MacOS X TODO

=Install Node-RED=

http://nodered.org/docs/hardware/raspberrypi

On Raspian Jessie : Node-RED is already installed (but you can update it with ) node-red-start pgrep node-red

The default directory is : ~/.node-red

Use                             to stop Node-RED

Use                            to start Node-RED again

Use     to autostart Node-RED at every boot

Use    to disable autostart on boot

Browse http://127.0.0.1:1880/

Now, you can write your own Node-RED flows and functions.

Install the admin cmdline tool

sudo npm install -g node-red-admin node-red-admin help

node-red-admin list
 * 1) installed modules

node-red-admin search serial node-red-admin search rfxcom node-red-admin search zwave node-red-admin search ble node-red-admin search bluetooth node-red-admin search sensortag
 * 1) search modules

node-red-admin search beaglebone node-red-admin search gpio

node-red-admin search mongodb node-red-admin search redis node-red-admin search cassandra node-red-admin search elastic node-red-admin search influxdb node-red-admin search kafka

Remark: node-red-admin install does not work ! Use

cd ~/.node-red sudo npm install -g node-red-node-redis sudo npm install -g node-red-contrib-kafka-consumer sudo npm install -g node-red-node-mongodb

Restart Node-Red

node-red-admin list

=Install extra nodes for Node-RED= Extra nodes are provided with the Node-RED community. There are listed here.

Serial for Geiger Counter


Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

Connect the Geiger counter to the host.

Check available serial ports (/dev/tty.usbserial* on MacOS X, ...) with.

Add a node serial "Geiger" with a Settings of 9600/8/N/1 and 'Split input into fixed lenghts of 1 chars'.

Add a node debug.

Connect "Geiger" to debug.

Deploy the flow.

The Geiger Counter sends a random sequence of 0 and 1.

Add a node function "Count Particles" with a flow-scoped variable (geiger/count):

Connect node "Geiger" to node "Count Particles".

Deploy the new flow.

Edit the node "Count Particles" and add the 2 following statements in order to display the count into the node's status.

Deploy the new flow.

Add a node inject "One minute timer" with a repeat interval of 1 minute.

Add a node function "Reset Particles Count" with a flow-scoped variable (geiger/count):

Connect node "One minute timer" to node "Reset Particles Count" and node "Reset Particles Count" to node debug.

Deploy the new flow.

The result is:

RFXCom for Oregon Weather Sensors
Install the rfxcom node (node-red-contrib-rfxcom) cd ~/.node-red npm install -g node-red-contrib-rfxcom

Restart Node-RED with.

Check the module in the list node-red-admin list | grep rfx

Connect the RFXCom receiver to the host.

Check available serial ports (/dev/tty.usbserial* on MacOS X, ...) with.

Add a node rfxcom-sensors "RFXCom" with the correct serial port.

Add a node debug display the full message (not only msg.payload).

Connect "RFXCom" to debug.

Deploy the flow.

The flow loos like that:



Serial for Arduino + Weather Shield
Plug the Weather Shield into the Arduino Leonardo.

Install the Arduino IDE on your host (link)

Install 2 extra libraries (Menu Sketch > Include Library > Add ZIP Library)
 * https://codeload.github.com/sparkfun/SparkFun_HTU21D_Breakout_Arduino_Library/zip/master
 * https://codeload.github.com/sparkfun/SparkFun_MPL3115A2_Breakout_Arduino_Library/zip/master

Select the right board (Arduino Leonardo) with the menu Tool

Load and unzip the Weather_Shield sketch bundle https://codeload.github.com/sparkfun/Weather_Shield/zip/master

Load the sketch Weather_Shield/firmware/Weather_Shield/Weather_Shield.ino into the Arduino Leonardo

Check the output with the menu Tool > Serial Monitor

The output looks like this: $,winddir=-1,windspeedmph=3.1,windgustmph=117.7,windgustdir=-1,windspdmph_avg2m=3.1,winddir_avg2m=0,windgustmph_10m=117.7,windgustdir_10m=-1,humidity=33.6,tempf=75.2,rainin=3.04,dailyrainin=3.04,pressure=98257.75,batt_lvl=4.28,light_lvl=0.28,#

Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports (/dev/tty.usbserial* on MacOS X, /dev/ttyUSB* ...) with.

Start Node-RED node-red -v

Add a node debug

Add a node function "Clean Data" with the following statements:

Add a node serial "Weather Station" with settings 9600/8/N/1 and split input on charracter /n

Connect them together like that

The flow is : [{"id":"40c61a94.428894","type":"serial-port","z":"65e2c142.626cf8","serialport":"/dev/tty.usbmodem1451","serialbaud":"9600","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false},{"id":"e0626709.612a1","type":"serial in","z":"65e2c142.626cf8","name":"Weather Station","serial":"40c61a94.428894","x":200.5,"y":486.5,"wires":"576a5389.61c4e4"},{"id":"bf78bff1.63e8a","type":"debug","z":"65e2c142.626cf8","name":"","active":true,"console":"false","complete":"payload","x":802.5,"y":488,"wires":[]},{"id":"576a5389.61c4e4","type":"function","z":"65e2c142.626cf8","name":"Clean data","func":"var m=msg.payload;\nvar i=m.indexOf(\",\")+1;\nmsg.payload=m.substr(i,m.lastIndexOf(\",\")-i);\nsetTimeout(function { node.status({}); }, 500)\nnode.status({fill:\"green\",shape:\"dot\",text:\"updated\"});\nreturn msg;","outputs":1,"noerr":0,"x":498.5,"y":486.5,"wires":"bf78bff1.63e8a"}]

Serial for LoRa Libelium


Install the Waspmote PRO IDE from Libelium on your host (link)

Load the sketch SX_02a_TX_LoRa on the LoRa Waspmote.

Check with the IDE Serial monitor the output of the Waspmote

Load the sketch SX_02b_RX_LoRa on the LoRa Dongle.

Check with the IDE Serial monitor the output of the USB Dongle

Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports ( on MacOS X,.

The RX boards receive LoRa frames and output the following lines on the console when the TX boards are powered : ... SX_02b example Semtech SX1272 module RX in LoRa

Setting configuration:

Setting Channel CH_10_868. state 0 Setting Header ON. state 0 Setting Mode '1'. state 0 Setting CRC ON. state 0 Setting Power to 'L'. state 0 Setting Node Address to '8'. state 0

Receiving:

Show packet received:

=
================= dest: 8 src: 2 packnum: 171 length: 26 retry: 0 payload (HEX): 546869735F69735F615F6E65775F6D657373616765 payload (string): This_is_a_new_message

=
================= ...

Create a new flow or start Node-RED

Add a node debug

Add a node function "Extract RX Data" with the following statements

Add a node serial "LoRa RX Modem" with settings 115200/8/N/1 and split input on character /n. On RPI, the port is.

Connect them together like that

The flow is : [{"id":"3124f278.d5cc86","type":"serial-port","z":"3cf8afc1.fe41a","serialport":"/dev/ttyUSB2","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false},{"id":"8ea9bd5f.7ffac8","type":"serial in","z":"3cf8afc1.fe41a","name":"","serial":"3124f278.d5cc86","x":123.5,"y":137.5,"wires":"a9a605c3.664548"},{"id":"32a2d4e0.54214c","type":"debug","z":"3cf8afc1.fe41a","name":"","active":true,"console":"false","complete":"false","x":495.5,"y":145,"wires":[]},{"id":"a9a605c3.664548","type":"function","z":"3cf8afc1.fe41a","name":"Extract RX Data","func":"var prefix=\"payload (HEX): \";\nvar m=msg.payload;\nvar i=m.indexOf(prefix);\nif(i===0) {\n   msg.payload=new Buffer(m.substr(prefix.length),\"HEX\");\n    setTimeout(function { node.status({}); }, 500)\n    node.status({fill:\"green\",shape:\"dot\",text:\"updated\"});\n    return msg;\n} else {\n    return null;\n}","outputs":1,"noerr":0,"x":320.5,"y":247.5,"wires":"32a2d4e0.54214c"}]

Serial for LoRa Nucleo


Install the serial node (node-red-node-serialport) npm install node-red-node-serialport or sudo npm install -g npm@2.x npm install node-red-node-serialport@0.0.5 if node.js prior to 4.x (ie v0.10.x and v0.12.x)

Restart Node-RED with.

Check the module in the list node-red-admin list | grep serial

Check available serial ports ( on MacOS X,.

Compile and load the SX1276Receiver program https://developer.mbed.org/users/donsez/code/SX1276Receiver/ on the Nucleo board.

For the receiver, set line 16 to

For the transmitter, set line 16 to

Remark: select the right board (F411 or L152 for instance).

The RX boards receive LoRa frames and output the following lines on the console when the TX boards are powered : ... >INFO RX modem=1 size=17 rssi=-41 snr=31 freq=868100000 bw=0 sf=12 cr=1 buffer=544553540004411d616263646566676869 RX;1;17;-41;31;868100000;0;12;1;544553540004411d616263646566676869 ...

Create a new flow or start Node-RED

Add a node debug

Add a node function "Extract RX Data" with the following statements

Add a node serial "LoRa RX Modem" with settings 115200/8/N/1 and split input on character /n. On RPI, the port is.

Connect them together like that

The flow is :

TODO

UDP for LoRaWAN
Add a node 'udp in' listening the port 8123.

Add a node 'function' "toString" with the following statement: msg.payload=msg.payload.toString; return msg;

Add a node 'debug'.

Connect them together.

Deploy the new flow.

Test the flow with the shell command echo -n -e '{"device"="123456","temperature"=37.2}' > /dev/udp/127.0.0.1/8123

The flow looks like that:

Add a node 'json' into the flow.

Now you can filter and send the output of a process to the flow througth an 'udp in' node.

TO BE CONTINUED

UDP for ESP8266 + PowerSensor


Install the Arduino IDE on your host (link)

Add the "ESP8266 Modules" support with the Tools menu > Board Manager.

Select "NodeMCU 0.9 (ESP8266-12 Module)" in the Tools menu.

Load the following sketch on the ESP8266.

Add a node 'udp in' listening the port 8123 with a Buffer output.

Add a node 'function' "toStr" with the following statement:

Add a node 'debug'.

Connect them together.

Deploy the new flow.

The flow looks like that:

Remark: you can test the flow with this shell command echo -n -e '{"device"="12348266","load"=10}' > /dev/udp/127.0.0.1/8123

The flow is: [{"id":"b32d523b.fd02d8","type":"udp in","z":"a994a424.14415","name":"","iface":"","port":"8123","ipv":"udp4","multicast":"false","group":"","datatype":"utf8","x":147,"y":94,"wires":"1658af1a.ebfd11"},{"id":"f732d1c6.93623","type":"debug","z":"a994a424.14415","name":"","active":true,"console":"false","complete":"payload","x":535,"y":94.5,"wires":[]},{"id":"1658af1a.ebfd11","type":"function","z":"a994a424.14415","name":"toStr","func":"setTimeout(function { node.status({}); }, 1000);\n// The shape property can be: ring or dot.\n// The fill property can be: red, green, yellow, blue or grey\nnode.status({fill:\"green\",shape:\"dot\",text:\"receiving\"}); \nmsg.payload=msg.payload.toString;\nreturn msg;","outputs":1,"noerr":0,"x":336,"y":94,"wires":"f732d1c6.93623"}]

Second part:

Plug the SCT-013 Current Clamp power shield into the ESP8266 Arduino board.

Compile and load the following sketch on the ESP8266 Arduino board.

Change the Node-RED to display the received data.

ACR122U NFC Reader
Plug the ACR122U NFC reader in the host and check it with

Install LibNFC (on Debian) sudo apt-get install libnfc-dev libnfc-bin libnfc-examples

sudo nfc-list

Launch nfc-poll and put a NFC card/tag on the reader sudo nfc-poll -v

The ID of the NFC card/tag is prefixed by the string.

Execute the following script :

while true do nfc-poll | grep "UID (NFCID1):" > /tmp/nfc.log done
 * 1) !/bin/sh

Into the Node-RED dashboard, create a new flow.

Add a node tail for reading the tail of the log file containing the card/tags identifiers.

Add a node debug.

Connect the node tail to the node debug.

Deploy the new flow.

Add a node function "Build message" with the following statements:

Connect the node tail to the node "Build message" and the node "Build message" to the node debug.

Deploy the new flow.

Edit the node "Build message" and add the 2 following statements in order to display the last tid into the node's status during one second.

Deploy the new flow.

Sigfox Patrol Man : Sigfox Areku Board + Adafruit NFC Shield
Install the libraries Areku and Adafruit_PN532 (https://github.com/PM2M-2016-NFCSigfox/pm2m/tree/master/arduino/libraries) into the Arduino IDE.

Compile and load the sketch into the Areku board. (https://github.com/PM2M-2016-NFCSigfox/pm2m/blob/master/arduino/rondesNfc/rondesNfc.ino)

Check with the IDE Serial monitor the USB output of the board and put a NFC tag on the PCB antenna of the Adafruit NFC Shield. Install and run the following Node.js script:

npm init npm install express --save sudo npm install mqtt --save cat package.json node server.js

Configure the callback http://52.50.55.74:3000/sigfox?data={data}&id={device}&time={time}&snr={snr}&station={station}&avgSnr={avgSnr}&rssi={rssi}&lat={lat}&lng={lng}&seqNumber={seqNumber}&duplicate={duplicate} for the device into the Sigfox backend. (account is required)

Create a flow with a MQTT in node and a debug node.

Parse and display the values sent by the Sigfox backend : device, time, duplicate, snr, station, data, avgSnr, lat, lng, rssi, seqNumber

TBC

Remarks:
 * The Sigfox modem can not send more than 144 messages due to the ETSI regulation.
 * A Sigfox message contains up to 12 bytes (authenticated but not encrypted)

LoRa Patrol Man : STM32 Nucleo + NFC Shield


See https://github.com/PM2M2016-STM32NUCLEO/M2M

Arduino 101 BLE


TODO

SensorTag2 BLE


TODO

Follow the instructions here : https://github.com/uwefassnacht/ti-sensor-tag-demo

=Install Mosquitto=

On MacOS X brew install mosquitto echo Test connectivity BROKER=localhost mosquitto_sub -h $BROKER -t '$SYS/#'

On Debian sudo apt-get install mosquitto sudo service mosquitto status sudo apt-get install mosquitto-clients echo Test connectivity BROKER=localhost mosquitto_sub -h $BROKER -t '$SYS/#'

Test local Mosquitto broker BROKER=localhost mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' & mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="123456","temperature"=37.2}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="345678","temperature"=23.2,"humidity"=50}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0002' -m '{"device"="acr122u","nfc"="04ddf0f9232580"}' sleep 1 pkill mosquitto_sub

Test public Mosquitto broker BROKER=test.mosquitto.org mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' & mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="123456","temperature"=37.2}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0001' -m '{"device"="345678","temperature"=23.2,"humidity"=50}' mosquitto_pub -d -h $BROKER -t 'iotdays/sensors/gw0002' -m '{"device"="acr122u","nfc"="04ddf0f9232580"}' pkill mosquitto_sub

More:
 * http://mosquitto.org/man/mosquitto_sub-1.html
 * http://mosquitto.org/man/mosquitto_pub-1.html

=Using MQTT for collecting sensors data=

In the flows defined above, add a node mqtt out with a new MQTT broker localhost:1883 and a topic iotdays/sensors/gwXXXX where gwXXXX is the name of the host.

Check the publishing with BROKER=localhost mosquitto_sub -d -h $BROKER -t 'iotdays/sensors/#' &

Create a new flow for subscribing messages from the Mosquitto MQTT broker.

Add a node debug for displaying the full message.

Add a node mqtt in with with a new MQTT broker localhost:1883 and a topic iotdays/sensors#

Connect the node mqtt in to the node debug.

Deploy the new flow.

=Install InfluxDB=

OS X (via Homebrew)
brew update brew install influxdb

Ubuntu & Debian (64-bit)
wget http://dl.influxdata.com/influxdb/releases/influxdb_0.12.2-1_amd64.deb sudo dpkg -i influxdb_0.12.2-1_amd64.deb

Ubuntu & Debian (ARM)
wget --no-check-certificate http://dl.influxdata.com/influxdb/releases/influxdb_0.12.2-1_armhf.deb sudo dpkg -i influxdb_0.12.2-1_armhf.deb

=Populate InfluxDB from Node-RED=

Launch the InfluxDB shell influx

Enter the following InfluxDB statements

In the Node-RED dashboard, create a new flow to collect sensors data from the MQTT server.

Install the Node-RED InfluxDB node.

Restart Node-RED.

Add a node mqtt in "MQTT Broker" for subscribing to the MQTT server (broker is test.mosquitto.org or localhost topic is iotdays/sensors/#)

Add a node function "Tranform into time series" to format the JSON messages into time series messages.

Add a node influxdb out "IoT Database" with the InfluxDB server.

Connect the node "MQTT Broker" to node "Tranform into time series" and to the node "Tranform into time series" to node "IoT Database".

The flow looks like than:

Launch the InfluxDB shell influx

Enter the following InfluxDB statements

Extra for Arduino and Weather Shield
Add a node influxdb out

Add a node function with the following statements:

Connect the 2 additionnal nodes like that:

The flow is: [{"id":"57400c2a.74f84c","type":"influxdb","z":"65e2c142.626cf8","hostname":"localhost","port":"8086","database":"iotdb","name":""},{"id":"40c61a94.428894","type":"serial-port","z":"65e2c142.626cf8","serialport":"/dev/tty.usbmodem1451","serialbaud":"9600","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false},{"id":"e0626709.612a1","type":"serial in","z":"65e2c142.626cf8","name":"Weather Station","serial":"40c61a94.428894","x":200.5,"y":486.5,"wires":"576a5389.61c4e4"},{"id":"bf78bff1.63e8a","type":"debug","z":"65e2c142.626cf8","name":"","active":true,"console":"false","complete":"payload","x":802.5,"y":488,"wires":[]},{"id":"576a5389.61c4e4","type":"function","z":"65e2c142.626cf8","name":"Clean data","func":"var m=msg.payload;\nvar i=m.indexOf(\",\")+1;\nmsg.payload=m.substr(i,m.lastIndexOf(\",\")-i);\nsetTimeout(function { node.status({}); }, 500)\nnode.status({fill:\"green\",shape:\"dot\",text:\"updated\"});\nreturn msg;","outputs":1,"noerr":0,"x":498.5,"y":486.5,"wires":"86b78cd4.e2d548","bf78bff1.63e8a"},{"id":"86b78cd4.e2d548","type":"function","z":"65e2c142.626cf8","name":"Format as InfluxDB line","func":"var result = {};\nmsg.payload.split(',').forEach(function(x){\n   var arr = x.split('=');\n    arr[1] && (result[arr[0]] = parseFloat(arr[1]));\n});\nmsg.payload=result;\nreturn msg;","outputs":1,"noerr":0,"x":667.5,"y":610,"wires":"1a959fff.cd4668","bf78bff1.63e8a"},{"id":"1a959fff.cd4668","type":"influxdb out","z":"65e2c142.626cf8","influxdb":"57400c2a.74f84c","name":"","measurement":"weather","x":908.5,"y":704,"wires":[]}]

Start the InfluxDB shell: influx

Create the database with the foolowing statements:

=Install Chronograf=

https://influxdata.com/downloads/#chronograf

OS X (via Homebrew)
brew update brew install homebrew/binary/chronograf

Ubuntu & Debian
wget https://s3.amazonaws.com/get.influxdb.org/chronograf/chronograf_0.12.0_amd64.deb sudo dpkg -i chronograf_0.12.0_amd64.deb

Remark: Chronograf configuration directory is ~/chronograf.db

=Visualize sensors data into the Chronograf dashboard=

TODO

Browse https://localhost:10000/

Configure the InfluxDB datasource



=Install Grafana (2.6)=

On Debian
wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.6.0_amd64.deb sudo dpkg -i xzvf grafana_2.6.0_amd64.deb sudo service grafana-server start

On Raspian Jessie
See https://github.com/heziegl/rpi-grafana

=Visualize sensors data into the Grafana dashboard=

Browse http://localhost:3000

Login with username:password admin:admin

Change your admin password (optional)

Create a new dashboard

Add a data source (InfluxDB 0.9) iotdb (http://localhost:8086 direct root:root)

Save the data source

Enable dashboard edition.

Add a row.

Add a graph in the row.

Edit the graph.

Edit the query for the graph for the data source.

Save the dashboard.

Duplicate the graph and change the measurement in the query

Duplicate the graph and aggregate (mean) all measurements of the same type (temperature, humidity).

Set the timeline and the refresh time.

Save the dashboard.

Move and resize the graphs into the dashboard.

Save the dashboard.

See Grafana

The result should look like that

=Install Apache Spark=

See Spark

=Process a realtime stream of sensors data=

TODO

=Annexes=