-
- Downloads
[SPARK-15814][SQL] Aggregator can return null result
## What changes were proposed in this pull request? It's similar to the bug fixed in https://github.com/apache/spark/pull/13425, we should consider null object and wrap the `CreateStruct` with `If` to do null check. This PR also improves the test framework to test the objects of `Dataset[T]` directly, instead of calling `toDF` and compare the rows. ## How was this patch tested? new test in `DatasetAggregatorSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #13553 from cloud-fan/agg-null.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala 6 additions, 1 deletion...rk/sql/execution/aggregate/TypedAggregateExpression.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala 21 additions, 2 deletions...t/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala 3 additions, 3 deletions...st/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 20 additions, 18 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 61 additions, 34 deletionssql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala 2 additions, 2 deletions...ache/spark/sql/execution/datasources/text/TextSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala 2 additions, 2 deletions.../org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/MemorySinkSuite.scala 2 additions, 2 deletions...cala/org/apache/spark/sql/streaming/MemorySinkSuite.scala
Loading
Please register or sign in to comment