Skip to content
Snippets Groups Projects
Commit da303526 authored by Mridul Muralidharan's avatar Mridul Muralidharan Committed by Sean Owen
Browse files

[SPARK-7323] [SPARK CORE] Use insertAll instead of insert while merging combiners in reducer

Author: Mridul Muralidharan <mridulm@yahoo-inc.com>

Closes #5862 from mridulm/optimize_aggregator and squashes the following commits:

61cf43a [Mridul Muralidharan] Use insertAll instead of insert - much more expensive to do it per tuple
parent 856a571e
No related branches found
No related tags found
No related merge requests found
......@@ -88,10 +88,7 @@ case class Aggregator[K, V, C] (
combiners.iterator
} else {
val combiners = new ExternalAppendOnlyMap[K, C, C](identity, mergeCombiners, mergeCombiners)
while (iter.hasNext) {
val pair = iter.next()
combiners.insert(pair._1, pair._2)
}
combiners.insertAll(iter)
// Update task metrics if context is not null
// TODO: Make context non-optional in a future release
Option(context).foreach { c =>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment