Skip to content
Snippets Groups Projects
Commit 95d28233 authored by Khanna's avatar Khanna
Browse files

update readme

parent 302bfa9d
Branches master
No related tags found
No related merge requests found
## Prerequisites:
# Prerequisites:
# Java 7+ installed on master & slave machines
# Scala 2+ installed on master & slave machines
# PySpark installed on master machine (`sudo pip install pyspark`)
# apache-spark project & streaming files cloned on each slave machine
### Java 7+ installed on master & slave machines
### Scala 2+ installed on master & slave machines
### PySpark installed on master machine (`sudo pip install pyspark`)
### apache-spark project & streaming files cloned on each slave machine
## Start up Spark cluster
# Start up Spark cluster
```
./sbin/start-all.sh
```
# This starts up VMs 2-10 as worker machines, cluster summary can be seen on http://MASTER-IP:8080/
### This starts up VMs 2-10 as worker machines, cluster summary can be seen on http://MASTER-IP:8080/
## Submit a word count job to the cluster
# Submit a word count job to the cluster
```
bin/spark-submit --master spark://MASTER-IP:7077 --deploy-mode client ~/apache-spark/python/wordcount.py ~/apache-spark/python/higgs-activity_time.txt
```
# Job summary can be seen on http://MASTER-IP:4040/
### Job summary can be seen on http://MASTER-IP:4040/
# Notes:
## Notes:
# 1. spark-env.sh file within conf/ should be modified as per job & master/slave requirements
# 2. /etc/hosts file on master & slave should have IP addresses resolved to the same hostnames within the slaves file in conf/
# 3. Project includes setup scripts/examples/driver code from Apache Spark
### 1. `spark-env.sh` file within `conf/` should be modified as per job & master/slave requirements
### 2. `/etc/hosts` file on master & slave should have IP addresses resolved to the same hostnames within the slaves file in `conf/`
### 3. Project includes setup scripts/examples/driver code from Apache Spark (https://github.com/apache/spark)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment