Skip to content
Snippets Groups Projects
Commit d3d2fd6a authored by Khanna's avatar Khanna
Browse files

updating README

parent 2121107b
No related branches found
No related tags found
No related merge requests found
## Prerequisites: # Prerequisites:
# Java 7+ installed on master & slave machines ## Java 7+ installed on master & slave machines
# Scala 2+ installed on master & slave machines ## Scala 2+ installed on master & slave machines
# PySpark installed on master machine (`sudo pip install pyspark`) ## PySpark installed on master machine (`sudo pip install pyspark`)
# apache-spark project & streaming files cloned on each slave machine ## apache-spark project & streaming files cloned on each slave machine
## Start up Spark cluster # Start up Spark cluster
``` ```
./sbin/start-all.sh ./sbin/start-all.sh
``` ```
# This starts up VMs 2-10 as worker machines, cluster summary can be seen on http://MASTER-IP:8080/ ## This starts up VMs 2-10 as worker machines, cluster summary can be seen on http://MASTER-IP:8080/
## Submit a word count job to the cluster # Submit a word count job to the cluster
``` ```
bin/spark-submit --master spark://MASTER-IP:7077 --deploy-mode client ~/apache-spark/python/wordcount.py ~/apache-spark/python/higgs-activity_time.txt bin/spark-submit --master spark://MASTER-IP:7077 --deploy-mode client ~/apache-spark/python/wordcount.py ~/apache-spark/python/higgs-activity_time.txt
``` ```
# Job summary can be seen on http://MASTER-IP:4040/ ## Job summary can be seen on http://MASTER-IP:4040/
# Notes: # Notes:
# 1. spark-env.sh file within conf/ should be modified as per job & master/slave requirements ## 1. `spark-env.sh` file within `conf/` should be modified as per job & master/slave requirements
# 2. /etc/hosts file on master & slave should have IP addresses resolved to the same hostnames within the slaves file in conf/ ## 2. `/etc/hosts` file on master & slave should have IP addresses resolved to the same hostnames within the slaves file in `conf/`
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment