Apache Kafka

Apache Kafka is Publish-Subscribe messaging rethought as a distributed commit log.

http://kafka.apache.org/

Clients en Perl, Python, Node.js, C, C++, Scala ...: https://cwiki.apache.org/confluence/display/KAFKA/Clients

=First steps with Kafka= see Quickstart

cd kafka

Launch Zookeeper ./bin/zookeeper-server-start.sh ./config/zookeeper.properties

Launch Kafka server ./bin/kafka-server-start.sh ./config/server.properties

Create a topic ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Launch Kafka console producer ./bin/kafka-console-producer.sh --broker-list localhost:2181 --topic test

Launch Kafka console consumer ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

Info on topic ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

Replicated topics
broker.id=1 port=9093 log.dir=/tmp/kafka-logs-1 broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2
 * 1) config/server-1.properties:
 * 1) config/server-2.properties:

Launch extra servers ./bin/kafka-server-start.sh config/server-1.properties & ./bin/kafka-server-start.sh config/server-2.properties &

Create a replicated topic ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Launch Kafka console producer ./bin/kafka-console-producer.sh --broker-list localhost:2181 --topic my-replicated-topic

Launch Kafka console consumer ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic

Produce and Consume messages with Node.js
TODO

Download master zipfile from https://github.com/SOHU-Co/kafka-node/ (or git clone https://github.com/SOHU-Co/kafka-node.git)

npm install kafka-node-master cd kafka-node-master

cd example node topics.js node producer.js

node consumer.js

Voir InfluxDB pour archiver les messages reçus dans une base InfluxDB.

Produce and Consume messages with Python
TODO

Stop all
Stop all ./bin/kafka-server-stop.sh ./bin/zookeeper-server-stop.sh

Extra
Launch Zookeeper shell ./bin/zookeeper-shell.sh localhost:2181