- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too

Implementing SPARK-878 for PySpark: adding zip and egg files to context ...

Two minor RDD refactoring

Moved shuffle serializer setting from a constructor parameter to a setSerializer method in various RDDs that involve shuffle operations.

Currying RDD.zipPartitions 

Implementing SPARK-878 for PySpark: adding zip and egg files to context and passing it down to workers which add these to their sys.path

Use the JSON formatter from Scala library and removed dependency on lift-json.

fixing typo in conf/slaves

It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).

This reverts commit 230ab272, reversing
changes made to 659553b2.

Updated json output to allow for display of worker state

Rename `memoryBytesToString` and `memoryMegabytesToString`

Fix PySpark unit tests on Python 2.6

These are used all over the place now and they are not specific to memory at all.

memoryBytesToString --> bytesToString
memoryMegabytesToString --> megabytesToString

Various UI improvements.

Ops teams need to ensure that the cluster is functional and performant.  Having to scrape the html source for worker state won't work reliably, and will be slow.  By exposing the state in the json output, ops teams are able to ensure a fully functional environment by querying for the json output and parsing for dead nodes.

A few small scheduler / job description changes.

1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.

2. Renamed SparkContext.setDescription to setCurrentJobDescription.

3. Throw an exception if the fair scheduler allocation file is invalid.

Adding GC Stats to TaskMetrics (and three small fixes)

Fixed 2 bugs in executor UI (incl. SPARK-877)

1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.

Set SPARK_CLASSPATH for maven repl tests

Update to Mesos 0.12.1

This fixes an error where empty cells are missing if a given task
has no shuffle read/write.