The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded.  The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.

I also made the gateway and jvm variables private.

This change results in ~3-4x performance improvement when running the
PySpark unit tests.

This patch alters the Python <-> executor protocol to pass on
exception data when they occur in user Python code.

Also, adds a line in doc explaining how to use.

cloudpickle runs into issues while pickling subclasses of AccumulatorParam,
which may be related to this Python issue:

    http://bugs.python.org/issue7689

This seems hard to fix and the ABCMeta wasn't necessary, so I removed it.

Fix minor documentation formatting issues.

This should avoid exceptions caused by existing
files with different contents.

I also removed some unused code.

PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.

This is a temporary change until we port takeSample().

Expand the PySpark programming guide.