Skip to content
Snippets Groups Projects
Commit 447ae2de authored by Ilya Ganelin's avatar Ilya Ganelin Committed by Andrew Or
Browse files

[SPARK-4569] Rename 'externalSorting' in Aggregator

Hi all - I've renamed the unhelpfully named variable and added a comment clarifying what's actually happening.

Author: Ilya Ganelin <ilya.ganelin@capitalone.com>

Closes #3666 from ilganeli/SPARK-4569B and squashes the following commits:

1810394 [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator
e2d2092 [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator
d7cefec [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator
5b3f39c [Ilya Ganelin] [SPARK-4569] Rename  in Aggregator
parent e230da18
No related branches found
No related tags found
No related merge requests found
...@@ -34,7 +34,9 @@ case class Aggregator[K, V, C] ( ...@@ -34,7 +34,9 @@ case class Aggregator[K, V, C] (
mergeValue: (C, V) => C, mergeValue: (C, V) => C,
mergeCombiners: (C, C) => C) { mergeCombiners: (C, C) => C) {
private val externalSorting = SparkEnv.get.conf.getBoolean("spark.shuffle.spill", true) // When spilling is enabled sorting will happen externally, but not necessarily with an
// ExternalSorter.
private val isSpillEnabled = SparkEnv.get.conf.getBoolean("spark.shuffle.spill", true)
@deprecated("use combineValuesByKey with TaskContext argument", "0.9.0") @deprecated("use combineValuesByKey with TaskContext argument", "0.9.0")
def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]]): Iterator[(K, C)] = def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]]): Iterator[(K, C)] =
...@@ -42,7 +44,7 @@ case class Aggregator[K, V, C] ( ...@@ -42,7 +44,7 @@ case class Aggregator[K, V, C] (
def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]], def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]],
context: TaskContext): Iterator[(K, C)] = { context: TaskContext): Iterator[(K, C)] = {
if (!externalSorting) { if (!isSpillEnabled) {
val combiners = new AppendOnlyMap[K,C] val combiners = new AppendOnlyMap[K,C]
var kv: Product2[K, V] = null var kv: Product2[K, V] = null
val update = (hadValue: Boolean, oldValue: C) => { val update = (hadValue: Boolean, oldValue: C) => {
...@@ -71,9 +73,9 @@ case class Aggregator[K, V, C] ( ...@@ -71,9 +73,9 @@ case class Aggregator[K, V, C] (
combineCombinersByKey(iter, null) combineCombinersByKey(iter, null)
def combineCombinersByKey(iter: Iterator[_ <: Product2[K, C]], context: TaskContext) def combineCombinersByKey(iter: Iterator[_ <: Product2[K, C]], context: TaskContext)
: Iterator[(K, C)] = : Iterator[(K, C)] =
{ {
if (!externalSorting) { if (!isSpillEnabled) {
val combiners = new AppendOnlyMap[K,C] val combiners = new AppendOnlyMap[K,C]
var kc: Product2[K, C] = null var kc: Product2[K, C] = null
val update = (hadValue: Boolean, oldValue: C) => { val update = (hadValue: Boolean, oldValue: C) => {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment