Skip to content
Snippets Groups Projects
Commit 3519b5e8 authored by Michael Armbrust's avatar Michael Armbrust
Browse files

[SPARK-2967][SQL] Follow-up: Also copy hash expressions in sort based shuffle fix.

Follow-up to #2066

Author: Michael Armbrust <michael@databricks.com>

Closes #2072 from marmbrus/sortShuffle and squashes the following commits:

2ff8114 [Michael Armbrust] Fix bug
parent 7e191fe2
No related branches found
No related tags found
No related merge requests found
......@@ -46,12 +46,15 @@ case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends Una
case HashPartitioning(expressions, numPartitions) =>
// TODO: Eliminate redundant expressions in grouping key and value.
val rdd = child.execute().mapPartitions { iter =>
@transient val hashExpressions =
newMutableProjection(expressions, child.output)()
if (sortBasedShuffleOn) {
@transient val hashExpressions =
newProjection(expressions, child.output)
iter.map(r => (hashExpressions(r), r.copy()))
} else {
@transient val hashExpressions =
newMutableProjection(expressions, child.output)()
val mutablePair = new MutablePair[Row, Row]()
iter.map(r => mutablePair.update(hashExpressions(r), r))
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment