-
- Downloads
[SPARK-15370][SQL] Fix count bug
# What changes were proposed in this pull request? This pull request fixes the COUNT bug in the `RewriteCorrelatedScalarSubquery` rule. After this change, the rule tests the expression at the root of the correlated subquery to determine whether the expression returns `NULL` on empty input. If the expression does not return `NULL`, the rule generates additional logic in the `Project` operator above the rewritten subquery. This additional logic intercepts `NULL` values coming from the outer join and replaces them with the value that the subquery's expression would return on empty input. This PR takes over https://github.com/apache/spark/pull/13155. It only fixes an issue with `Literal` construction and style issues. All credits should go frreiss. # How was this patch tested? Added regression tests to cover all branches of the updated rule (see changes to `SubquerySuite`). Ran all existing automated regression tests after merging with latest trunk. Author: frreiss <frreiss@us.ibm.com> Author: Herman van Hovell <hvanhovell@databricks.com> Closes #13629 from hvanhovell/SPARK-15370-cleanup.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala 5 additions, 2 deletions...rg/apache/spark/sql/catalyst/expressions/predicates.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 201 additions, 20 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala 81 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
Loading
Please register or sign in to comment