Smarter take/limit implementation.

Fix spacing so java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS

Refactor FairSchedulableBuilder

1. Configuration can be read from classpath if not set explicitly.
2. Add missing close handler.

Fix PR926 local properties issues in Spark Streaming like scenarios

Add "org.apache." prefix to packages in spark-class

After unit tests, clear port properties unconditionally

Lacking this, the if/case statements never trigger on Spark 0.8.0+.

In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port
and spark.hostPort, assuming that they will be cleared by
LocalSparkContext. However, the test never sets sc, so it remains null,
causing LocalSparkContext to skip clearing these properties. Subsequent
tests therefore fail with java.net.BindException: "Address already in
use".

This commit makes LocalSparkContext clear the properties even if sc is
null.

Fix issue with spark_ec2 seeing empty security groups

Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.

Fix typo in Maven build docs

Bumping Mesos version to 0.13.0

Explain yarn.version in Maven build docs

Use different Hadoop version for YARN artifacts.

This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.

Changed localProperties to use ThreadLocal (not DynamicVariable).

Updated Spark on Mesos documentation.

The fact that DynamicVariable uses an InheritableThreadLocal
can cause problems where the properties end up being shared
across threads in certain circumstances.

Add explicit jets3t dependency, which is excluded in hadoop-client