-
- Downloads
SPARK-1255: Allow user to pass Serializer object instead of class name for shuffle.
This is more general than simply passing a string name and leaves more room for performance optimizations. Note that this is technically an API breaking change in the following two ways: 1. The shuffle serializer specification in ShuffleDependency now require an object instead of a String (of the class name), but I suspect nobody else in this world has used this API other than me in GraphX and Shark. 2. Serializer's in Spark from now on are required to be serializable. Author: Reynold Xin <rxin@apache.org> Closes #149 from rxin/serializer and squashes the following commits: 5acaccd [Reynold Xin] Properly call serializer's constructors. 2a8d75a [Reynold Xin] Added more documentation for the serializer option in ShuffleDependency. 7420185 [Reynold Xin] Allow user to pass Serializer object instead of class name for shuffle.
Showing
- core/src/main/scala/org/apache/spark/Dependency.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/Dependency.scala
- core/src/main/scala/org/apache/spark/ShuffleFetcher.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/ShuffleFetcher.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 14 additions, 10 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala 9 additions, 9 deletionscore/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
- core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala 4 additions, 3 deletions...rc/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
- core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala 7 additions, 6 deletionscore/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala
- core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala 10 additions, 10 deletionscore/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala 2 additions, 1 deletion...ain/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
- core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala 18 additions, 9 deletions...in/scala/org/apache/spark/serializer/JavaSerializer.scala
- core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala 10 additions, 6 deletions...in/scala/org/apache/spark/serializer/KryoSerializer.scala
- core/src/main/scala/org/apache/spark/serializer/Serializer.scala 13 additions, 3 deletions...c/main/scala/org/apache/spark/serializer/Serializer.scala
- core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala 0 additions, 75 deletions...scala/org/apache/spark/serializer/SerializerManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 1 addition, 1 deletion.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 6 additions, 3 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala 1 addition, 1 deletion...c/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/MessageToPartition.scala 6 additions, 6 deletions...ala/org/apache/spark/graphx/impl/MessageToPartition.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/Serializers.scala 7 additions, 7 deletions...main/scala/org/apache/spark/graphx/impl/Serializers.scala
- graphx/src/test/scala/org/apache/spark/graphx/SerializerSuite.scala 12 additions, 18 deletions.../test/scala/org/apache/spark/graphx/SerializerSuite.scala
Loading
Please register or sign in to comment