-
- Downloads
[SPARK-8992][SQL] Add pivot to dataframe api
This adds a pivot method to the dataframe api. Following the lead of cube and rollup this adds a Pivot operator that is translated into an Aggregate by the analyzer. Currently the syntax is like: ~~courseSales.pivot(Seq($"year"), $"course", Seq("dotNET", "Java"), sum($"earnings"))~~ ~~Would we be interested in the following syntax also/alternatively? and~~ courseSales.groupBy($"year").pivot($"course", "dotNET", "Java").agg(sum($"earnings")) //or courseSales.groupBy($"year").pivot($"course").agg(sum($"earnings")) Later we can add it to `SQLParser`, but as Hive doesn't support it we cant add it there, right? ~~Also what would be the suggested Java friendly method signature for this?~~ Author: Andrew Ray <ray.andrew@gmail.com> Closes #7841 from aray/sql-pivot.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 42 additions, 0 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala 14 additions, 0 deletions...che/spark/sql/catalyst/plans/logical/basicOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/GroupedData.scala 93 additions, 10 deletions...ore/src/main/scala/org/apache/spark/sql/GroupedData.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 7 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFramePivotSuite.scala 87 additions, 0 deletions...test/scala/org/apache/spark/sql/DataFramePivotSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestData.scala 12 additions, 0 deletions...rc/test/scala/org/apache/spark/sql/test/SQLTestData.scala
Loading
Please register or sign in to comment