-
- Downloads
[SPARK-4397][Core] Reorganize 'implicit's to improve the API convenience
This PR moved `implicit`s to `package object` and `companion object` to enable the Scala compiler search them automatically without explicit importing. It should not break any API. A test project for backforward compatibility is [here](https://github.com/zsxwing/SPARK-4397-Backforward-Compatibility). It proves the codes compiled with Spark 1.1.0 can run with this PR. To summarize, the changes are: * Deprecated the old implicit conversion functions: this preserves binary compatibility for code compiled against earlier versions of Spark. * Removed "implicit" from them so they are just normal functions: this made sure the compiler doesn't get confused and warn about multiple implicits in scope. * Created new implicit functions in package rdd object, which is part of the scope that scalac will search when looking for implicit conversions on various RDD objects. The disadvantage is there are duplicated codes in SparkContext for backforward compatibility. Author: zsxwing <zsxwing@gmail.com> Closes #3262 from zsxwing/SPARK-4397 and squashes the following commits: fc30314 [zsxwing] Update the comments 9c27aff [zsxwing] Move implicit functions to object RDD and forward old functions to new implicit ones directly 2b5f5a4 [zsxwing] Comments for the deprecated functions 52353de [zsxwing] Remove private[spark] from object WritableConverter 34641d4 [zsxwing] Move ImplicitSuite to org.apache.sparktest 7266218 [zsxwing] Add comments to warn the duplicate codes in SparkContext 185c12f [zsxwing] Remove simpleWritableConverter from SparkContext 3bdcae2 [zsxwing] Move WritableConverter implicits to object WritableConverter 9b73188 [zsxwing] Fix the code style issue 3ac4f07 [zsxwing] Add license header 1eda9e4 [zsxwing] Reorganize 'implicit's to improve the API convenience
Showing
- core/src/main/scala/org/apache/spark/Accumulators.scala 30 additions, 0 deletionscore/src/main/scala/org/apache/spark/Accumulators.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 120 additions, 41 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 1 addition, 1 deletion...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 30 additions, 0 deletionscore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/test/scala/org/apache/spark/SparkContextSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/SparkContextSuite.scala
- core/src/test/scala/org/apache/sparktest/ImplicitSuite.scala 128 additions, 0 deletionscore/src/test/scala/org/apache/sparktest/ImplicitSuite.scala
Loading
Please register or sign in to comment