-
- Downloads
[SPARK-12706] [SQL] grouping() and grouping_id()
Grouping() returns a column is aggregated or not, grouping_id() returns the aggregation levels. grouping()/grouping_id() could be used with window function, but does not work in having/sort clause, will be fixed by another PR. The GROUPING__ID/grouping_id() in Hive is wrong (according to docs), we also did it wrongly, this PR change that to match the behavior in most databases (also the docs of Hive). Author: Davies Liu <davies@databricks.com> Closes #10677 from davies/grouping.
Showing
- python/pyspark/sql/dataframe.py 11 additions, 11 deletionspython/pyspark/sql/dataframe.py
- python/pyspark/sql/functions.py 44 additions, 0 deletionspython/pyspark/sql/functions.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala 8 additions, 5 deletions...main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 38 additions, 10 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala 5 additions, 0 deletions...rg/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 2 additions, 0 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala 0 additions, 1 deletion...spark/sql/catalyst/expressions/aggregate/interfaces.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala 23 additions, 0 deletions.../org/apache/spark/sql/catalyst/expressions/grouping.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala 1 addition, 1 deletion...che/spark/sql/catalyst/plans/logical/basicOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 46 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala 44 additions, 0 deletions.../scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 50 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala 5 additions, 3 deletions...che/spark/sql/hive/execution/HiveCompatibilitySuite.scala
- sql/hive/src/test/resources/golden/SPARK-8976 Wrong Result for CUBE #1-0-63b61fb3f0e74226001ad279be440864 6 additions, 6 deletions...ong Result for CUBE #1-0-63b61fb3f0e74226001ad279be440864
- sql/hive/src/test/resources/golden/SPARK-8976 Wrong Result for Rollup #1-0-a78e3dbf242f240249e36b3d3fd0926a 6 additions, 6 deletions...g Result for Rollup #1-0-a78e3dbf242f240249e36b3d3fd0926a
- sql/hive/src/test/resources/golden/SPARK-8976 Wrong Result for Rollup #2-0-bf180c9d1a18f61b9d9f31bb0115cf89 10 additions, 10 deletions...g Result for Rollup #2-0-bf180c9d1a18f61b9d9f31bb0115cf89
- sql/hive/src/test/resources/golden/SPARK-8976 Wrong Result for Rollup #3-0-9257085d123728730be96b6d9fbb84ce 10 additions, 10 deletions...g Result for Rollup #3-0-9257085d123728730be96b6d9fbb84ce
Loading
Please register or sign in to comment