-
- Downloads
[SPARK-6367][SQL] Use the proper data type for those expressions that are...
[SPARK-6367][SQL] Use the proper data type for those expressions that are hijacking existing data types. This PR adds internal UDTs for expressions that are hijacking existing data types. The following UDTs are added: * `HyperLogLogUDT` (`BinaryType` as the SQL type) for `ApproxCountDistinctPartition` * `OpenHashSetUDT` (`ArrayType` as the SQL type) for `CollectHashSet`, `NewSet`, `AddItemToSet`, and `CombineSets`. I am also adding more unit tests for aggregation with code gen enabled. JIRA: https://issues.apache.org/jira/browse/SPARK-6367 Author: Yin Huai <yhuai@databricks.com> Closes #5094 from yhuai/expressionType and squashes the following commits: 8bcd11a [Yin Huai] Return types. 61a1d66 [Yin Huai] Merge remote-tracking branch 'upstream/master' into expressionType e8b4599 [Yin Huai] Merge remote-tracking branch 'upstream/master' into expressionType 2753156 [Yin Huai] Ignore aggregations having sum functions for now. b5eb259 [Yin Huai] Case object for HyperLogLog type. 00ebdbd [Yin Huai] deserialize/serialize. 54b87ae [Yin Huai] Add UDTs for expressions that return HyperLogLog and OpenHashSet.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala 21 additions, 3 deletions...rg/apache/spark/sql/catalyst/expressions/aggregates.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 2 additions, 2 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/sets.scala 30 additions, 5 deletions...cala/org/apache/spark/sql/catalyst/expressions/sets.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala 8 additions, 4 deletions...a/org/apache/spark/sql/execution/GeneratedAggregate.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 7 additions, 5 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala 23 additions, 1 deletion...est/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala
Loading
Please register or sign in to comment