-
- Downloads
[SPARK-18711][SQL] should disable subexpression elimination for LambdaVariable
## What changes were proposed in this pull request? This is kind of a long-standing bug, it's hidden until https://github.com/apache/spark/pull/15780 , which may add `AssertNotNull` on top of `LambdaVariable` and thus enables subexpression elimination. However, subexpression elimination will evaluate the common expressions at the beginning, which is invalid for `LambdaVariable`. `LambdaVariable` usually represents loop variable, which can't be evaluated ahead of the loop. This PR skips expressions containing `LambdaVariable` when doing subexpression elimination. ## How was this patch tested? updated test in `DatasetAggregatorSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #16143 from cloud-fan/aggregator.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala 5 additions, 1 deletion...park/sql/catalyst/expressions/EquivalentExpressions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala 4 additions, 4 deletions...t/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
Loading
Please register or sign in to comment