-
- Downloads
[SPARK-17849][SQL] Fix NPE problem when using grouping sets
## What changes were proposed in this pull request? Prior this pr, the following code would cause an NPE: `case class point(a:String, b:String, c:String, d: Int)` `val data = Seq( point("1","2","3", 1), point("4","5","6", 1), point("7","8","9", 1) )` `sc.parallelize(data).toDF().registerTempTable("table")` `spark.sql("select a, b, c, count(d) from table group by a, b, c GROUPING SETS ((a)) ").show()` The reason is that when the grouping_id() behavior was changed in #10677, some code (which should be changed) was left out. Take the above code for example, prior #10677, the bit mask for set "(a)" was `001`, while after #10677 the bit mask was changed to `011`. However, the `nonNullBitmask` was not changed accordingly. This pr will fix this problem. ## How was this patch tested? add integration tests Author: wangyang <wangyang@haizhi.com> Closes #15416 from yangw1234/groupingid. (cherry picked from commit fb0d6081) Signed-off-by:Herman van Hovell <hvanhovell@databricks.com>
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 7 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql 17 additions, 0 deletions...core/src/test/resources/sql-tests/inputs/grouping_set.sql
- sql/core/src/test/resources/sql-tests/results/grouping_set.sql.out 42 additions, 0 deletions...src/test/resources/sql-tests/results/grouping_set.sql.out
Loading
Please register or sign in to comment