-
- Downloads
[SPARK-14042][CORE] Add custom coalescer support
## What changes were proposed in this pull request? This PR adds support for specifying an optional custom coalescer to the `coalesce()` method. Currently I have only added this feature to the `RDD` interface, and once we sort out the details we can proceed with adding this feature to the other APIs (`Dataset` etc.) ## How was this patch tested? Added a unit test for this functionality. /cc rxin (per our discussion on the mailing list) Author: Nezih Yigitbasi <nyigitbasi@netflix.com> Closes #11865 from nezihyigitbasi/custom_coalesce_policy.
Showing
- core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala 49 additions, 50 deletionscore/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 6 additions, 3 deletionscore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/rdd/coalesce-public.scala 52 additions, 0 deletions...src/main/scala/org/apache/spark/rdd/coalesce-public.scala
- core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala 98 additions, 1 deletioncore/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
- project/MimaExcludes.scala 4 additions, 0 deletionsproject/MimaExcludes.scala
Loading
Please register or sign in to comment