Apache Kafka

Apache Kafka is Publish-Subscribe messaging rethought as a distributed commit log.

http://kafka.apache.org/

Clients en Perl, Python, Node.js, C, C++, Scala ...: https://cwiki.apache.org/confluence/display/KAFKA/Clients

=First steps with Kafka= see Quickstart

cd kafka

Launch Zookeeper ./bin/zookeeper-server-start.sh ./config/zookeeper.properties

Launch Kafka server ./bin/kafka-server-start.sh ./config/server.properties

Create a topic ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Launch Kafka console producer ./bin/kafka-console-producer.sh --broker-list localhost:2181 --topic test

Launch Kafka console consumer ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

Info on topic ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

Replicated topics
broker.id=1 port=9093 log.dir=/tmp/kafka-logs-1 broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2
 * 1) config/server-1.properties:
 * 1) config/server-2.properties:

Launch extra servers ./bin/kafka-server-start.sh config/server-1.properties & ./bin/kafka-server-start.sh config/server-2.properties &

Create a replicated topic ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Launch Kafka console producer ./bin/kafka-console-producer.sh --broker-list localhost:2181 --topic my-replicated-topic

Launch Kafka console consumer ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic

Produce and Consume messages with Node.js
TODO

Download master zipfile from https://github.com/SOHU-Co/kafka-node/ (or git clone https://github.com/SOHU-Co/kafka-node.git)

npm install kafka-node-master cd kafka-node-master

cd example node topics.js node producer.js

node consumer.js

Produce and Consume messages with Python
TODO

Stop all
Stop all ./bin/kafka-server-stop.sh ./bin/zookeeper-server-stop.sh

Extra
Launch Zookeeper shell ./bin/zookeeper-shell.sh localhost:2181