Signed-off-by: shane-huang <shengsheng.huang@intel.com>

Signed-off-by: shane-huang <shengsheng.huang@intel.com>

Refactor FairSchedulableBuilder

1. Configuration can be read from classpath if not set explicitly.
2. Add missing close handler.

Fix PR926 local properties issues in Spark Streaming like scenarios

Add "org.apache." prefix to packages in spark-class

After unit tests, clear port properties unconditionally

Lacking this, the if/case statements never trigger on Spark 0.8.0+.

In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port
and spark.hostPort, assuming that they will be cleared by
LocalSparkContext. However, the test never sets sc, so it remains null,
causing LocalSparkContext to skip clearing these properties. Subsequent
tests therefore fail with java.net.BindException: "Address already in
use".

This commit makes LocalSparkContext clear the properties even if sc is
null.

Fix issue with spark_ec2 seeing empty security groups

Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.

Fix typo in Maven build docs

Bumping Mesos version to 0.13.0

Explain yarn.version in Maven build docs

Use different Hadoop version for YARN artifacts.

This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.

Changed localProperties to use ThreadLocal (not DynamicVariable).

Updated Spark on Mesos documentation.

The fact that DynamicVariable uses an InheritableThreadLocal
can cause problems where the properties end up being shared
across threads in certain circumstances.

Add explicit jets3t dependency, which is excluded in hadoop-client

Change default port number from 3030 to 4030.

SPARK-894 - Not all WebUI fields delivered VIA JSON

fix run-example script

Fix HDFS access bug with assembly build.

Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549

there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.