Skip to content
Snippets Groups Projects
Commit ff7201cf authored by Patrick Wendell's avatar Patrick Wendell
Browse files

Merge pull request #461 from pwendell/master


Use renamed shuffle spill config in CoGroupedRDD.scala

This one got missed when it was renamed.
(cherry picked from commit aa981e4e)

Signed-off-by: default avatarPatrick Wendell <pwendell@gmail.com>
parent 7b0d5a5f
No related branches found
No related tags found
No related merge requests found
...@@ -106,7 +106,7 @@ class CoGroupedRDD[K](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]], part: ...@@ -106,7 +106,7 @@ class CoGroupedRDD[K](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]], part:
override def compute(s: Partition, context: TaskContext): Iterator[(K, CoGroupCombiner)] = { override def compute(s: Partition, context: TaskContext): Iterator[(K, CoGroupCombiner)] = {
val sparkConf = SparkEnv.get.conf val sparkConf = SparkEnv.get.conf
val externalSorting = sparkConf.getBoolean("spark.shuffle.externalSorting", true) val externalSorting = sparkConf.getBoolean("spark.shuffle.spill", true)
val split = s.asInstanceOf[CoGroupPartition] val split = s.asInstanceOf[CoGroupPartition]
val numRdds = split.deps.size val numRdds = split.deps.size
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment