InfluxDB: Difference between revisions

From air
Jump to navigation Jump to search
 
(66 intermediate revisions by 2 users not shown)
Line 1: Line 1:
La version courante est la version 0.10

Vers la page de la version 0.8 [[InfluxDB 0.8]]


http://influxdb.com/
http://influxdb.com/


https://github.com/influxdb/influxdb
https://github.com/influxdb/influxdb


open-source distributed time series database with no external dependencies.
open-source distributed time series database for storing metrics, events, and analytics.

(metrics, events, and analytics)


developped in [[Go]] language
developped in [[Go]] language

* retention policy : data is automatically deleted after the retention period
* sharding :
* replication :


Voir https://speakerdeck.com/pauldix/introducing-influxdb-an-open-source-distributed-time-series-database
Voir https://speakerdeck.com/pauldix/introducing-influxdb-an-open-source-distributed-time-series-database
Line 20: Line 27:
http://influxdb.com/download/
http://influxdb.com/download/


==On Linux (Debian)==
On OS X
<pre>
<pre>
# for 64-bit systems
brew update
wget https://dl.influxdata.com/influxdb/releases/influxdb_0.13.0_amd64.deb
brew install influxdb
sudo dpkg -i influxdb_0.13.0_amd64.deb
sudo service influxdb start
sudo service influxdb status
</pre>
</pre>


==On OS X==
Démarrage
<pre>
<pre>
brew update
influxdb -config=/usr/local/etc/influxdb.conf
brew install influxdb
</pre>
</pre>


==With [[Docker]]==
Naviguez sur http://localhost:8083/
See https://hub.docker.com/r/tutum/influxdb/
username : root & password : root


=Démarrage=
Pensez à changer ces valeurs quand vous mettez influxdb en production


==On Linux Debian==
=Premiers Pas=


Configurer Influxdb si besoin en éditant /etc/influxdb/influxdb.conf
==Via l'interface Web==
Depuis l'interface web, Créez une base : mydb

Depuis ''Database > Explore Data''

Depuis ''write point'', Ajoutez un point dans la série temporelle ''log_lines''


<pre>
<pre>
sudo service influxdb start
{ "line":"here's some useful log info from paul@influx.com", "like":1, "star": 5 }
ps wwwax | grep influxdb
</pre>
</pre>

puis un autre
L'arrêt se fait avec
<pre>
<pre>
sudo service influxdb stop
{ "line":"here's another useful log info from paul@influx.com", "like":2, "star": 4 }
</pre>
</pre>

puis un autre
==On MacOS X==

<pre>
<pre>
sudo /etc/init.d/influxdb start
{ "line":"here's some useful log info from didier@donsez.com", "like":1, "star": 5 }
</pre>
</pre>


Naviguez sur http://localhost:8083/
Exécutez les requêtes suivantes depuis ''read point'':
username : root & password : root


Pensez à changer ces valeurs dans /usr/local/etc/influxdb.conf quand vous mettez influxdb en production


Le fichier de configuration peut être généré au moyen de :
La liste des séries temporelles
<pre>
list series
</pre>
<pre>
<pre>
/opt/influxdb/influxd config > /etc/influxdb/influxdb.generated.conf
select * from /.*/ limit 1
</pre>
</pre>


Remarque: les interfaces pour UDP, Graphite, ... peuvent être activées.


'''Remarque: en production, il convient de sécuriser les ports (ie HTTPS, ...) si le serveur n'est pas derrière un firewall'''.
Les points de la série log_lines
<pre>
select line from log_lines
</pre>


=[https://influxdb.com/docs/v0.9/concepts/schema_and_data_layout.html Schema Design]=
Le schéma de la base est constituée de time series contenant des datapoints.

Un datapoint est un n-uplets:
<pre>
<pre>
timeserie,tag=value,tag=value, measure=value,measure=value timestamp
select like, star from log_lines
</pre>
</pre>


Les datapoints sont indexés sur le timestamp (primaire) et sur les tags (secondaires)


=Premiers Pas=
Les deux points les plus récents
<pre>
select * from log_lines limit 2;
</pre>


==Via le shell==
Les points de la série log_lines dont la colonne line contient paul@influx.com
<pre>
<pre>
influx
select line from log_lines where line =~ /paul@influx.com/
influx -host localhost
</pre>
</pre>


Créer la base
Les points de la série log_lines dont la colonne like est supérieure à 1
<pre>
<pre>
CREATE DATABASE mydb
select line from log_lines where like > 1
SHOW DATABASES
USE mydb
</pre>
</pre>


Insérer des points
Les points de la série log_lines des 24 dernières heures
<pre>
<pre>
INSERT cpu,host=serverA,region=us_west value=0.64
select * from log_lines
INSERT cpu,host=serverA,region=us_east value=0.70
where time > now() - 24h
INSERT cpu,host=serverA,region=us_west value=0.70
</pre>
INSERT cpu,host=serverA,region=us_east value=0.90


INSERT payment,device=mobile,product=Notepad,method=credit billed=33,licenses=3i 1434067467100293230


INSERT stock,symbol=AAPL bid=127.46,ask=127.48
Agrégat temporel (les fonctions d'agrégat sont : count(), min(), max(), mean(), mode(), median(), distinct(), percentile(), histogram(), derivative(), sum(), stddev(), first(), last()).

<pre>
INSERT temperature,machine=unit42,type=assembly external=25,internal=37 1434067467000000000
select sum(like) as number_of_likes, mean(star) as mean_of_star from log_lines
group by time(1m)
where time > now() - 1d
</pre>
</pre>

Consulter la base
<pre>
<pre>
show series
select star, sum(like) as number_of_likes from log_lines
group by star, time(1m)
where time > now() - 1d
</pre>


SELECT * FROM cpu
Remplissage des intervalles vides de points (avec fill())
SELECT * FROM cpu WHERE value >= 0.7
<pre>
SELECT * FROM cpu LIMIT 1
select count(like) from events
SELECT value from cpu WHERE time > now() - 7d
group by time(1h) fill(0)
where time > now() - 24h
SELECT mean(value) FROM cpu WHERE time > now() - 1h GROUP BY time(10m);

SELECT * FROM temperature
</pre>
</pre>


Consulter la base en continue (''[https://influxdb.com/docs/v0.9/query_language/continuous_queries.html Continuous Queries]'')

Fusion (merge) de séries
<pre>
<pre>
TODO
select count(type) from user_events merge admin_events group by time(10m)

# Create a CQ to sample the 95% value from 5 minute buckets of the response_times measurement
CREATE CONTINUOUS QUERY response_times_percentile ON mydb BEGIN
SELECT percentile(value, 95) INTO "response_times.percentiles.5m.95" FROM response_times GROUP BY time(5m)
END
</pre>
</pre>




Lister et supprimer des requêtes en continu
Attention, cette requête n'est pas un produit cartésien (SQL)
<pre>
<pre>
SHOW CONTINUOUS QUERIES
select * from log_lines, log_cpu;

DROP CONTINUOUS QUERY response_times_percentile ON mydb
</pre>
</pre>


Changer la rétention de la base mydb (ie les données ne sont conservées qu'un jour)
Jointure entre 2 séries
<pre>
<pre>
ALTER RETENTION POLICY monitor ON mydb DURATION 1d REPLICATION 1 DEFAULT
select errors_per_minute.value / page_views_per_minute.value as error_rate_per_minute.
from errors_per_minute
inner join page_views_per_minute
</pre>
</pre>



Création d'un [http://influxdb.com/docs/v0.8/api/continuous_queries.html Requêtes continues]
Consulter les statistiques
<pre>
<pre>
SHOW STATS
select count(star) from log_lines group by time(10m), star
into log_lines.count_per_star.10m
</pre>
</pre>



Consulter le diagnostique
<pre>
<pre>
SHOW DIAGNOSTICS
select * from log_lines.count_per_star.10m
</pre>
</pre>


Consulter le cluster
<pre>
<pre>
SHOW SERVERS
drop continuous query <id>
</pre>
</pre>




Consulter la base interne '''_internal'''

Supprimez les points de toutes les séries temporelles dont la date est antérieure de 24 heures.
<pre>
delete from /.*/ where time < now() - 24h
select * from /.*/
</pre>

Suppression d'une série
<pre>
<pre>
USE _internal
drop series log_lines
show measurements
show series limit 10
select * from httpd limit 10
select * from runtime limit 10
</pre>
</pre>


Line 169: Line 186:


<pre>
<pre>
curl -G 'http://localhost:8086/db/mydb/series?u=root&p=root&pretty=true' --data-urlencode "q=select * from log_lines"
curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode \
"q=SELECT value FROM cpu WHERE region='us_west'"
</pre>
</pre>


Requêtes multiples
===Administration===
create a database
<pre>
<pre>
curl -X POST 'http://localhost:8086/db?u=root&p=root' -d '{"name": "site_development"}'
curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode \
"q=SELECT value FROM cpu WHERE region='us_west';SELECT count(value) FROM cpu WHERE region='us_west'"
</pre>
</pre>


Affichage en seconde
drop a database
<pre>
<pre>
curl -X DELETE 'http://localhost:8086/db/site_development?u=root&p=root'
curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" \
--data-urlencode "epoch=s" --data-urlencode "q=SELECT value FROM cpu WHERE region='us_west'"
</pre>
</pre>


Affichage d'un 'chunk' de 20000
Autre : http://influxdb.com/docs/v0.8/api/administration.html
<pre>curl

curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" \
==Dashboard avec Influga==
--data-urlencode "chunk_size=20000" --data-urlencode "q=SELECT * FROM cpu"
Installez
<pre>
sudo npm install -g influga
</pre>
</pre>


==Via l'interface Web==
Depuis l'interface web http://localhost:8083/, Créez une base : mydb


=Clustering & Replication=
Influxdb utilise le protocole [[Raft]] pour l'élection d'un leader : il faut configurer au minimum 3 machines avec des noeuds '''meta'''. Remarque : une machine pour être un noeud '''meta''' seul, un noeud '''data''' seul, ou bien les deux.
Le nombre de noeuds '''data''' dépend du niveau de réplication désiré.




https://docs.influxdata.com/influxdb/v0.10/guides/clustering/
==Dashboard avec [[Grafana]]==
http://influxdb.com/docs/v0.7/ui/grafana.html


Vérifier la configuration sous le shell influx
Téléchargez Grafana


Modifiez config.example.js avec
<pre>
<pre>
SHOW SERVERS
datasources: {
'eu-metrics': {
type: 'influxdb',
url: 'http://localhost:8086/db/mydb',
username: 'test',
password: 'test',
},
'grafana': {
type: 'influxdb',
url: 'http://localhost:8086/db/grafana',
username: 'test',
password: 'test',
grafanaDB: true
},
},
</pre>
</pre>


=Backup=
et Sauvegardez dans config.js
[https://docs.influxdata.com/influxdb/v0.10/administration/backup_and_restore/ Manuel]

<pre>
influxd backup -database telegraf -retention default -since 2016-02-01T00:00:00Z /tmp/backup/telegraf-2016-02-01.db.backup
</pre>

=Restore=
<pre>
sudo service influxdb stop
influxd restore -database telegraf -datadir /var/lib/influxdb/data /tmp/backup/telegraf-2016-02-01.db.backup
sudo service influxdb start
influx -execute 'show databases'
</pre>


==Projets connexes==
Ouvrez index.html
==[[Telegraf]]==
https://github.com/influxdb/telegraf
''agent written in Go for collecting metrics from the system it's running on, or from other services, and writing them into InfluxDB. ''


==[[Chronograf]]==
=Premières Requêtes=
''Chronograf is a single binary web application that you can deploy behind your firewall to do ad hoc exploration of your time series data in InfluxDB.''
([https://influxdata.com/time-series-platform/chronograf/ lien])


Installation et utilisation détaillées [[Chronograf|ici]].


==[[Node.js]]==
==[[Kapacitor]]==
Kapacitor is a data processing engine. It can process both stream (subscribe realtime) and batch (bulk query) data from InfluxDB. Kapacitor lets you define custom logic to process alerts with dynamic thresholds, match metrics for patterns, compute statistical anomalies, etc. ([https://influxdata.com/time-series-platform/kapacitor/ lien])
* https://github.com/bencevans/node-influx


=Extra=
==[[Node-RED]]==
* [[Jmxtrans]]
* https://github.com/node-red/node-red/issues/462
* [[Grafana]]

Latest revision as of 15:46, 1 September 2016

La version courante est la version 0.10

Vers la page de la version 0.8 InfluxDB 0.8


http://influxdb.com/

https://github.com/influxdb/influxdb

open-source distributed time series database for storing metrics, events, and analytics.

developped in Go language

  • retention policy : data is automatically deleted after the retention period
  • sharding :
  • replication :

Voir https://speakerdeck.com/pauldix/introducing-influxdb-an-open-source-distributed-time-series-database

Demo sur http://play.influxdb.com/ (source code)

InfluxDB @ AIR

Installation

http://influxdb.com/download/

On Linux (Debian)

# for 64-bit systems
wget https://dl.influxdata.com/influxdb/releases/influxdb_0.13.0_amd64.deb
sudo dpkg -i influxdb_0.13.0_amd64.deb 
sudo service influxdb start
sudo service influxdb status

On OS X

brew update
brew install influxdb

With Docker

See https://hub.docker.com/r/tutum/influxdb/

Démarrage

On Linux Debian

Configurer Influxdb si besoin en éditant /etc/influxdb/influxdb.conf 

sudo service influxdb start
ps wwwax | grep influxdb

L'arrêt se fait avec 

sudo service influxdb stop

On MacOS X

sudo /etc/init.d/influxdb start

Naviguez sur http://localhost:8083/ username : root & password : root

Pensez à changer ces valeurs dans /usr/local/etc/influxdb.conf quand vous mettez influxdb en production

Le fichier de configuration peut être généré au moyen de : 

/opt/influxdb/influxd config > /etc/influxdb/influxdb.generated.conf

Remarque: les interfaces pour UDP, Graphite, ... peuvent être activées.

Remarque: en production, il convient de sécuriser les ports (ie HTTPS, ...) si le serveur n'est pas derrière un firewall.

Schema Design

Le schéma de la base est constituée de time series contenant des datapoints.

Un datapoint est un n-uplets: 

timeserie,tag=value,tag=value, measure=value,measure=value timestamp

Les datapoints sont indexés sur le timestamp (primaire) et sur les tags (secondaires)

Premiers Pas

Via le shell

influx
influx -host localhost

Créer la base 

CREATE DATABASE mydb
SHOW DATABASES
USE mydb

Insérer des points 

INSERT cpu,host=serverA,region=us_west value=0.64
INSERT cpu,host=serverA,region=us_east value=0.70
INSERT cpu,host=serverA,region=us_west value=0.70
INSERT cpu,host=serverA,region=us_east value=0.90

INSERT payment,device=mobile,product=Notepad,method=credit billed=33,licenses=3i 1434067467100293230

INSERT stock,symbol=AAPL bid=127.46,ask=127.48

INSERT temperature,machine=unit42,type=assembly external=25,internal=37 1434067467000000000

Consulter la base 

show series

SELECT * FROM cpu
SELECT * FROM cpu WHERE value >= 0.7
SELECT * FROM cpu LIMIT 1
SELECT value from cpu WHERE time > now() - 7d
SELECT mean(value) FROM cpu WHERE time > now() - 1h GROUP BY time(10m);

SELECT * FROM temperature

Consulter la base en continue (Continuous Queries) 

TODO

# Create a CQ to sample the 95% value from 5 minute buckets of the response_times measurement
CREATE CONTINUOUS QUERY response_times_percentile ON mydb BEGIN
  SELECT percentile(value, 95) INTO "response_times.percentiles.5m.95" FROM response_times GROUP BY time(5m)
END


Lister et supprimer des requêtes en continu 

SHOW CONTINUOUS QUERIES

DROP CONTINUOUS QUERY response_times_percentile ON mydb

Changer la rétention de la base mydb (ie les données ne sont conservées qu'un jour) 

ALTER RETENTION POLICY monitor ON mydb DURATION 1d REPLICATION 1 DEFAULT


Consulter les statistiques 

SHOW STATS


Consulter le diagnostique 

SHOW DIAGNOSTICS

Consulter le cluster 

SHOW SERVERS


Consulter la base interne _internal 

USE _internal
show measurements
show series limit 10
select * from httpd limit 10
select * from runtime limit 10

Via l'interface REST

curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode \
 "q=SELECT value FROM cpu WHERE region='us_west'"

Requêtes multiples 

curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode \
 "q=SELECT value FROM cpu WHERE region='us_west';SELECT count(value) FROM cpu WHERE region='us_west'"

Affichage en seconde 

curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" \
 --data-urlencode "epoch=s" --data-urlencode "q=SELECT value FROM cpu WHERE region='us_west'"

Affichage d'un 'chunk' de 20000 

curl
curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" \
 --data-urlencode "chunk_size=20000" --data-urlencode "q=SELECT * FROM cpu"

Via l'interface Web

Depuis l'interface web http://localhost:8083/, Créez une base : mydb

Clustering & Replication

Influxdb utilise le protocole Raft pour l'élection d'un leader : il faut configurer au minimum 3 machines avec des noeuds meta. Remarque : une machine pour être un noeud meta seul, un noeud data seul, ou bien les deux. Le nombre de noeuds data dépend du niveau de réplication désiré.


https://docs.influxdata.com/influxdb/v0.10/guides/clustering/

Vérifier la configuration sous le shell influx 

SHOW SERVERS

Backup

Manuel 

influxd backup -database telegraf -retention default -since 2016-02-01T00:00:00Z /tmp/backup/telegraf-2016-02-01.db.backup

Restore

sudo service influxdb stop
influxd restore -database telegraf -datadir /var/lib/influxdb/data /tmp/backup/telegraf-2016-02-01.db.backup
sudo service influxdb start
influx -execute 'show databases'

Projets connexes

Telegraf

https://github.com/influxdb/telegraf agent written in Go for collecting metrics from the system it's running on, or from other services, and writing them into InfluxDB.

Chronograf

Chronograf is a single binary web application that you can deploy behind your firewall to do ad hoc exploration of your time series data in InfluxDB. (lien)

Installation et utilisation détaillées ici.

Kapacitor

Kapacitor is a data processing engine. It can process both stream (subscribe realtime) and batch (bulk query) data from InfluxDB. Kapacitor lets you define custom logic to process alerts with dynamic thresholds, match metrics for patterns, compute statistical anomalies, etc. (lien)

Extra

Retrieved from "https://air.imag.fr/index.php?title=InfluxDB&oldid=30363"