Apache Kafka - Revision history

Donsez: /* Extra */

2017-08-01T09:47:57Z

Extra

← Older revision		Revision as of 09:47, 1 August 2017
Line 126:		Line 126:
	</pre>		</pre>

			==Quickstart with [[Docker]]==
			TODO

	=Un peu plus=		=Un peu plus=

Donsez: /* Schema Registry */

2017-08-01T09:39:45Z

Schema Registry

← Older revision		Revision as of 09:39, 1 August 2017
Line 170:		Line 170:
	[[Apache Avro]] est le serialisateur de prédilection de l’écosystême Kafka.		[[Apache Avro]] est le serialisateur de prédilection de l’écosystême Kafka.

	Confluent propose un registry des schemas Avro utilisés.		Confluent propose un registry des schemas [[Avro\|Apache Avro]] utilisés.

	Le Schema Registry est en dehors du projet Apache Kafka.		Le Schema Registry est en dehors du projet Apache Kafka.

Donsez: /* Kafka Streams */

2017-08-01T09:36:03Z

Kafka Streams

← Older revision		Revision as of 09:36, 1 August 2017
Line 163:		Line 163:
	==[[Kafka Streams]]==		==[[Kafka Streams]]==

	Canevas de [[Event Stream Processing]] basé sur [[Kafka]] et [[Kafka Connect]]		Canevas de [[Event Stream Processing]] basé sur [[Apache Kafka\|Kafka]] et [[Kafka Connect]]

	==Schema Registry==		==Schema Registry==

Donsez: /* Produce and Consume messages with Node-RED */

2017-08-01T09:35:22Z

Produce and Consume messages with Node-RED

← Older revision		Revision as of 09:35, 1 August 2017
Line 106:		Line 106:
	==Produce and Consume messages with [[Node-RED]]==		==Produce and Consume messages with [[Node-RED]]==
	TODO		TODO

			https://www.npmjs.com/package/node-red-contrib-kafka-node

			<pre>
			npm install -g node-red-contrib-kafka-node
			</pre>

	==Stop all==		==Stop all==

Donsez: /* Produce and Consume messages with Python */

2017-08-01T09:33:51Z

Produce and Consume messages with Python

← Older revision		Revision as of 09:33, 1 August 2017
Line 102:		Line 102:

	==Produce and Consume messages with [[Python]]==		==Produce and Consume messages with [[Python]]==
			TODO

			==Produce and Consume messages with [[Node-RED]]==
	TODO		TODO

Donsez: /* Kafka Connect */

2017-08-01T09:32:41Z

Kafka Connect

← Older revision		Revision as of 09:32, 1 August 2017
Line 134:		Line 134:
	https://drive.google.com/file/d/0B_0n2CoDWpWQbkVsSUZ2SC1aQkk/view?usp=sharing		https://drive.google.com/file/d/0B_0n2CoDWpWQbkVsSUZ2SC1aQkk/view?usp=sharing

	permet de connecter Kafka à des sources et puits d'info : File, MySQL, ELK, HDFS, …		permet de connecter Kafka à des sources et puits d'info : File, MySQL, ELK, HDFS, … Des connectors sont implémentables au moyen des interfaces Connector et Task.

	''it provides out-of-the-box features like configuration management, offset storage, parallelization, error handling, support for different data types, and standard management REST APIs.'' (Chapitre 7 du livre OReilly “Kafka : The Definitive Guide”)		''it provides out-of-the-box features like configuration management, offset storage, parallelization, error handling, support for different data types, and standard management REST APIs.'' (Chapitre 7 du livre OReilly “Kafka : The Definitive Guide”)

	* https://github.com/confluentinc/kafka-connect-jdbc		* https://github.com/confluentinc/kafka-connect-jdbc
	* https://github.com/confluentinc/kafka-connect-storage-cloud		* https://github.com/confluentinc/kafka-connect-storage-cloud
Line 144:		Line 143:
	Confluent recense et liste des connectors open-source et commerciaux : https://www.confluent.io/product/connectors/		Confluent recense et liste des connectors open-source et commerciaux : https://www.confluent.io/product/connectors/

	Stream Reactor : Un gros projet opensource de connecteurs (MQTT, InfluxDB, Azure DocumentDB, MongoDB, Blockchain…) écrits en Scala. Utilise un DSL de requêtage http://docs.datamountaineer.com/en/latest/kcql.html#kcql		Stream Reactor : Un gros projet opensource de connecteurs ([[MQTT]], [[InfluxDB]], [[Azure DocumentDB]], [[MongoDB]], Blockchain…) écrits en [[Scala]]. Utilise un DSL de requêtage [http://docs.datamountaineer.com/en/latest/kcql.html#kcql KCQL].

	* https://github.com/datamountaineer/stream-reactor		* https://github.com/datamountaineer/stream-reactor
	* https://github.com/datamountaineer/kafka-connect-tools		* https://github.com/datamountaineer/kafka-connect-tools

Donsez: /* Kafka Streams */

2017-08-01T09:30:04Z

Kafka Streams

← Older revision		Revision as of 09:30, 1 August 2017
Line 156:		Line 156:
	==[[Kafka Streams]]==		==[[Kafka Streams]]==

	Canevas de [[Stream Processing]] basé sur [[Kafka]] et [[Kafka Connect]]		Canevas de [[Event Stream Processing]] basé sur [[Kafka]] et [[Kafka Connect]]

	==Schema Registry==		==Schema Registry==

Donsez: /* Kafka Streams */

2017-08-01T09:28:39Z

Kafka Streams

← Older revision		Revision as of 09:28, 1 August 2017
Line 156:		Line 156:
	==[[Kafka Streams]]==		==[[Kafka Streams]]==

			Canevas de [[Stream Processing]] basé sur [[Kafka]] et [[Kafka Connect]]

	==Schema Registry==		==Schema Registry==

Donsez: /* Un peu plus */

2017-08-01T09:27:18Z

Un peu plus

← Older revision		Revision as of 09:27, 1 August 2017
Line 119:		Line 119:

	=Un peu plus=		=Un peu plus=
			==Livre==
⚫	[[Kafka Streams]]
			livre OReilly “Kafka : The Definitive Guide”, https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/

			==Kafka UI==
			http://docs.datamountaineer.com/en/latest/ui.html#install


			==[[Kafka REST Proxy]]==
			provides a RESTful interface to a Kafka cluster. The API is not documented with [[Swagger]] (ie [[OpenAPI]]).
			* https://github.com/confluentinc/kafka-rest
			* https://hub.docker.com/r/confluentinc/cp-kafka-rest/

			==[[Kafka Connect]]==
			https://drive.google.com/file/d/0B_0n2CoDWpWQbkVsSUZ2SC1aQkk/view?usp=sharing

			permet de connecter Kafka à des sources et puits d'info : File, MySQL, ELK, HDFS, …

			''it provides out-of-the-box features like configuration management, offset storage, parallelization, error handling, support for different data types, and standard management REST APIs.'' (Chapitre 7 du livre OReilly “Kafka : The Definitive Guide”)

			* https://github.com/confluentinc/kafka-connect-jdbc
			* https://github.com/confluentinc/kafka-connect-storage-cloud
			* https://github.com/confluentinc/kafka-connect-hdfs

			Confluent recense et liste des connectors open-source et commerciaux : https://www.confluent.io/product/connectors/

			Stream Reactor : Un gros projet opensource de connecteurs (MQTT, InfluxDB, Azure DocumentDB, MongoDB, Blockchain…) écrits en Scala. Utilise un DSL de requêtage http://docs.datamountaineer.com/en/latest/kcql.html#kcql

			* https://github.com/datamountaineer/stream-reactor
			* https://github.com/datamountaineer/kafka-connect-tools

			===Cluster Replication with Kafka===
			* Kafka MirrorMaker
			* Confluent Replicator
			* Uber uReplicator https://github.com/uber/uReplicator

		⚫	==[[Kafka Streams]]==


			==Schema Registry==
			http://docs.confluent.io/current/schema-registry/docs/intro.html

			[[Apache Avro]] est le serialisateur de prédilection de l’écosystême Kafka.

			Confluent propose un registry des schemas Avro utilisés.

			Le Schema Registry est en dehors du projet Apache Kafka.

Donsez: /* Extra */

2016-09-13T12:24:42Z

Extra

← Older revision		Revision as of 12:24, 13 September 2016
Line 116:		Line 116:
	./bin/zookeeper-shell.sh localhost:2181		./bin/zookeeper-shell.sh localhost:2181
	</pre>		</pre>


			=Un peu plus=
			[[Kafka Streams]]