Skip to content
Snippets Groups Projects
  1. Nov 10, 2013
    • Josh Rosen's avatar
      Add custom serializer support to PySpark. · cbb7f04a
      Josh Rosen authored
      For now, this only adds MarshalSerializer, but it lays the groundwork
      for other supporting custom serializers.  Many of these mechanisms
      can also be used to support deserialization of different data formats
      sent by Java, such as data encoded by MsgPack.
      
      This also fixes a bug in SparkContext.union().
      cbb7f04a
  2. Aug 16, 2013
  3. Aug 14, 2013
  4. Jul 16, 2013
  5. Jun 21, 2013
  6. Feb 01, 2013
    • Josh Rosen's avatar
      Do not launch JavaGateways on workers (SPARK-674). · 9cc6ff9c
      Josh Rosen authored
      The problem was that the gateway was being initialized whenever the
      pyspark.context module was loaded.  The fix uses lazy initialization
      that occurs only when SparkContext instances are actually constructed.
      
      I also made the gateway and jvm variables private.
      
      This change results in ~3-4x performance improvement when running the
      PySpark unit tests.
      9cc6ff9c
    • Josh Rosen's avatar
      Fix stdout redirection in PySpark. · 57b64d0d
      Josh Rosen authored
      57b64d0d
  7. Jan 25, 2013
  8. Jan 23, 2013
  9. Jan 22, 2013
  10. Jan 20, 2013
Loading