Skip to content
Snippets Groups Projects
Commit d6ed5abf authored by aniketbhatnagar's avatar aniketbhatnagar Committed by Andrew Or
Browse files

SPARK-3639 | Removed settings master in examples


This patch removes setting of master as local in Kinesis examples so that users can set it using submit-job.

Author: aniketbhatnagar <aniket.bhatnagar@gmail.com>

Closes #2536 from aniketbhatnagar/Kinesis-Examples-Master-Unset and squashes the following commits:

c9723ac [aniketbhatnagar] Merge remote-tracking branch 'origin/Kinesis-Examples-Master-Unset' into Kinesis-Examples-Master-Unset
fec8ead [aniketbhatnagar] SPARK-3639 | Removed settings master in examples
31cdc59 [aniketbhatnagar] SPARK-3639 | Removed settings master in examples

(cherry picked from commit d16e161d)
Signed-off-by: default avatarAndrew Or <andrewor14@gmail.com>
parent 505ed6ba
No related branches found
No related tags found
No related merge requests found
......@@ -71,6 +71,9 @@ import com.google.common.collect.Lists;
* org.apache.spark.examples.streaming.JavaKinesisWordCountASL mySparkStream \
* https://kinesis.us-east-1.amazonaws.com
*
* Note that number of workers/threads should be 1 more than the number of receivers.
* This leaves one thread available for actually processing the data.
*
* There is a companion helper class called KinesisWordCountProducerASL which puts dummy data
* onto the Kinesis stream.
* Usage instructions for KinesisWordCountProducerASL are provided in the class definition.
......@@ -114,12 +117,8 @@ public final class JavaKinesisWordCountASL { // needs to be public for access fr
/* In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard */
int numStreams = numShards;
/* Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
int numSparkThreads = numStreams + 1;
/* Setup the Spark config. */
SparkConf sparkConfig = new SparkConf().setAppName("KinesisWordCount").setMaster(
"local[" + numSparkThreads + "]");
SparkConf sparkConfig = new SparkConf().setAppName("KinesisWordCount");
/* Kinesis checkpoint interval. Same as batchInterval for this example. */
Duration checkpointInterval = batchInterval;
......
......@@ -65,6 +65,10 @@ import org.apache.log4j.Level
* org.apache.spark.examples.streaming.KinesisWordCountASL mySparkStream \
* https://kinesis.us-east-1.amazonaws.com
*
*
* Note that number of workers/threads should be 1 more than the number of receivers.
* This leaves one thread available for actually processing the data.
*
* There is a companion helper class below called KinesisWordCountProducerASL which puts
* dummy data onto the Kinesis stream.
* Usage instructions for KinesisWordCountProducerASL are provided in that class definition.
......@@ -97,17 +101,10 @@ private object KinesisWordCountASL extends Logging {
/* In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard. */
val numStreams = numShards
/*
* numSparkThreads should be 1 more thread than the number of receivers.
* This leaves one thread available for actually processing the data.
*/
val numSparkThreads = numStreams + 1
/* Setup the and SparkConfig and StreamingContext */
/* Spark Streaming batch interval */
val batchInterval = Milliseconds(2000)
val batchInterval = Milliseconds(2000)
val sparkConfig = new SparkConf().setAppName("KinesisWordCount")
.setMaster(s"local[$numSparkThreads]")
val ssc = new StreamingContext(sparkConfig, batchInterval)
/* Kinesis checkpoint interval. Same as batchInterval for this example. */
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment