Skip to content
Snippets Groups Projects
Commit e63a86ab authored by Daoyuan Wang's avatar Daoyuan Wang Committed by Michael Armbrust
Browse files

[SPARK-6872] [SQL] add copy in external sort

We need add copy before call externalsort.

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes #5481 from adrian-wang/extsort and squashes the following commits:

9611586 [Daoyuan Wang] fix bug in external sort
parent 2a55cb41
No related branches found
No related tags found
No related merge requests found
...@@ -194,7 +194,7 @@ case class ExternalSort( ...@@ -194,7 +194,7 @@ case class ExternalSort(
child.execute().mapPartitions( { iterator => child.execute().mapPartitions( { iterator =>
val ordering = newOrdering(sortOrder, child.output) val ordering = newOrdering(sortOrder, child.output)
val sorter = new ExternalSorter[Row, Null, Row](ordering = Some(ordering)) val sorter = new ExternalSorter[Row, Null, Row](ordering = Some(ordering))
sorter.insertAll(iterator.map(r => (r, null))) sorter.insertAll(iterator.map(r => (r.copy, null)))
val baseIterator = sorter.iterator.map(_._1) val baseIterator = sorter.iterator.map(_._1)
// TODO(marmbrus): The complex type signature below thwarts inference for no reason. // TODO(marmbrus): The complex type signature below thwarts inference for no reason.
CompletionIterator[Row, Iterator[Row]](baseIterator, sorter.stop()) CompletionIterator[Row, Iterator[Row]](baseIterator, sorter.stop())
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment