Skip to content
Snippets Groups Projects
Commit 4b4b50c9 authored by Michael Armbrust's avatar Michael Armbrust
Browse files

[SQL] Don't shuffle code generated rows

When sort based shuffle and code gen are on we were trying to ship the code generated rows during a shuffle.  This doesn't work because the classes don't exist on the other side.  Instead we now copy into a generic row before shipping.

Author: Michael Armbrust <michael@databricks.com>

Closes #3263 from marmbrus/aggCodeGen and squashes the following commits:

f6ba8cf [Michael Armbrust] fix and test
parent f805025e
No related branches found
No related tags found
No related merge requests found
...@@ -47,8 +47,8 @@ case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends Una ...@@ -47,8 +47,8 @@ case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends Una
// TODO: Eliminate redundant expressions in grouping key and value. // TODO: Eliminate redundant expressions in grouping key and value.
val rdd = if (sortBasedShuffleOn) { val rdd = if (sortBasedShuffleOn) {
child.execute().mapPartitions { iter => child.execute().mapPartitions { iter =>
val hashExpressions = newProjection(expressions, child.output) val hashExpressions = newMutableProjection(expressions, child.output)()
iter.map(r => (hashExpressions(r), r.copy())) iter.map(r => (hashExpressions(r).copy(), r.copy()))
} }
} else { } else {
child.execute().mapPartitions { iter => child.execute().mapPartitions { iter =>
......
...@@ -72,6 +72,13 @@ class SQLQuerySuite extends QueryTest with BeforeAndAfterAll { ...@@ -72,6 +72,13 @@ class SQLQuerySuite extends QueryTest with BeforeAndAfterAll {
2.5) 2.5)
} }
test("aggregation with codegen") {
val originalValue = codegenEnabled
setConf(SQLConf.CODEGEN_ENABLED, "true")
sql("SELECT key FROM testData GROUP BY key").collect()
setConf(SQLConf.CODEGEN_ENABLED, originalValue.toString)
}
test("SPARK-3176 Added Parser of SQL LAST()") { test("SPARK-3176 Added Parser of SQL LAST()") {
checkAnswer( checkAnswer(
sql("SELECT LAST(n) FROM lowerCaseData"), sql("SELECT LAST(n) FROM lowerCaseData"),
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment