Reduce the amount of duplicate logging Akka does to stdout.

Given we have Akka logging go through SLF4j to log4j, we don't need
all the extra noise of Akka's stdout logger that is supposedly only
used during Akka init time but seems to continue logging lots of
noisy network events that we either don't care about or are in the
log4j logs anyway.

See:

http://doc.akka.io/docs/akka/2.0/general/configuration.html

    # Log level for the very basic logger activated during AkkaApplication startup
    # Options: ERROR, WARNING, INFO, DEBUG
    # stdout-loglevel = "WARNING"

These operations used to wait for all the results to be available in an
array on the driver program before merging them. They now merge values
incrementally as they arrive.

Add more private declarations.

Stop BlockManagers metadataCleaner.

Use spark.local.dir for PySpark temp files (SPARK-580).

Do not launch JavaGateways on workers (SPARK-674).

The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded.  The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.

I also made the gateway and jvm variables private.

This change results in ~3-4x performance improvement when running the
PySpark unit tests.

Changed PartitionPruningRDD's split to make sure it returns the correct split index.

Fix stdout redirection in PySpark.

SPARK-673: Capture and re-throw Python exceptions

Remove activation of profiles by default

This patch alters the Python <-> executor protocol to pass on
exception data when they occur in user Python code.

split index.

See the discussion at https://github.com/mesos/spark/pull/355 for why
default profile activation is a problem.

Minor improvements to PySpark docs

Also, adds a line in doc explaining how to use.

It's nicer if all the commands you need are made explicit.

Remember ConnectionManagerId used to initiate SendingConnections

Make ExecutorIDs include SlaveIDs when running Mesos

Include message and exitStatus if availalbe.

the Mesos ExecutorID as a Spark ExecutorID.

This prevents ConnectionManager from getting confused if a machine
has multiple host names and the one getHostName() finds happens
not to be the one that was passed from, e.g., the BlockManagerMaster.

Conflicts:
	core/src/main/scala/spark/deploy/master/Master.scala

Add RDD.toDebugString.