Apache Flink: Difference between revisions
| (8 intermediate revisions by the same user not shown) | |||
| Line 17: | Line 17: | ||
==Local Execution== |
==Local Execution== |
||
Terminal 1: start Flink |
|||
<pre> |
<pre> |
||
cd $FLINK_HOME |
cd $FLINK_HOME |
||
| Line 24: | Line 24: | ||
Open the UI http://localhost:8081/#/overview |
Open the UI http://localhost:8081/#/overview |
||
[[Image:flink-ui.png|thumb|right| |
[[Image:flink-ui.png|thumb|right|500px|Flink UI]] |
||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
Run the SocketWindowWordCount example ([https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java source]). |
Run the SocketWindowWordCount example ([https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java source]). |
||
Terminal |
Terminal 2: Start netcat |
||
<pre> |
<pre> |
||
nc -l 9000 |
nc -l 9000 |
||
| Line 41: | Line 34: | ||
Submit the Flink program: |
Terminal 3: Submit the Flink program: |
||
<pre> |
<pre> |
||
| ⚫ | |||
bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000 |
bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000 |
||
</pre> |
</pre> |
||
Terminal |
Terminal 2: Add words in netcat input |
||
<pre> |
<pre> |
||
lorem ipsum |
lorem ipsum |
||
| Line 55: | Line 49: | ||
Terminal |
Terminal 4: |
||
<pre> |
<pre> |
||
cd $FLINK_HOME |
|||
tail -f log/flink-*-jobmanager-*.out |
tail -f log/flink-*-jobmanager-*.out |
||
</pre> |
</pre> |
||
| Line 62: | Line 57: | ||
Terminal |
Terminal 1: stop Flink |
||
<pre> |
<pre> |
||
cd $FLINK_HOME |
|||
bin/stop-local.sh |
bin/stop-local.sh |
||
</pre> |
</pre> |
||
==Cluster execution== |
|||
| ⚫ | |||
==Shell== |
|||
| ⚫ | |||
cd $FLINK_HOME |
|||
| ⚫ | |||
| ⚫ | |||
TBC |
|||
==Cluster execution== |
==Cluster execution== |
||
https://ci.apache.org/projects/flink/flink-docs-release-1.1/quickstart/setup_quickstart.html#cluster-setup |
|||
===Amazon AWS EC2=== |
|||
| ⚫ | |||
Install AWS CLI |
|||
<pre> |
|||
sudo apt-get install awscli |
|||
aws help |
|||
</pre> |
|||
Configure CLI with AWS credential ([http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html link]) |
|||
<pre> |
|||
aws configure |
|||
</pre> |
|||
''NB : credential file is ~/.aws/credentials and config file is ~/.aws/config'' |
|||
Create an cluster on AWS EMR (Elastic Map Reduce) in your AWS console ([https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.html link]). |
|||
[[Image:aws-emr-ui.png|400px|right|thumb|EMR Dashboard]] |
|||
The nodes of the EMR cluster are listed in the AWS EC2 panel of your AWS console. |
|||
Connect to Master node |
|||
<pre> |
|||
ssh -i ~/.ssh/awskey.pem hadoop@ec2-52-12-35-67.eu-west-1.compute.amazonaws.com |
|||
</pre> |
|||
Latest revision as of 07:25, 16 September 2016
Apache Flink® is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
Getting started
Installation
wget http://www.apache.org/dyn/closer.lua/flink/flink-1.1.2/flink-1.1.2-bin-hadoop27-scala_2.11.tgz tar xf flink-1.1.2-bin-hadoop27-scala_2.11.tgz FLINK_HOME=~/flink-1.1.2 cd $FLINK_HOME ls bin ls examples
Local Execution
Terminal 1: start Flink
cd $FLINK_HOME bin/start-local.sh
Open the UI http://localhost:8081/#/overview
Run the SocketWindowWordCount example (source).
Terminal 2: Start netcat
nc -l 9000
Terminal 3: Submit the Flink program:
cd $FLINK_HOME bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
Terminal 2: Add words in netcat input
lorem ipsum ipsum ipsum ipsum bye
Terminal 4:
cd $FLINK_HOME tail -f log/flink-*-jobmanager-*.out
Terminal 1: stop Flink
cd $FLINK_HOME bin/stop-local.sh
Shell
cd $FLINK_HOME bin/start-scala-shell.sh local
TBC
Cluster execution
Amazon AWS EMR
Install AWS CLI
sudo apt-get install awscli aws help
Configure CLI with AWS credential (link)
aws configure
NB : credential file is ~/.aws/credentials and config file is ~/.aws/config
Create an cluster on AWS EMR (Elastic Map Reduce) in your AWS console (link).
The nodes of the EMR cluster are listed in the AWS EC2 panel of your AWS console.
Connect to Master node
ssh -i ~/.ssh/awskey.pem hadoop@ec2-52-12-35-67.eu-west-1.compute.amazonaws.com