-
- Downloads
[SPARK-17076][SQL] Cardinality estimation for join based on basic column statistics
## What changes were proposed in this pull request? Support cardinality estimation and stats propagation for all join types. Limitations: - For inner/outer joins without any equal condition, we estimate it like cartesian product. - For left semi/anti joins, since we can't apply the heuristics for inner join to it, for now we just propagate the statistics from left side. We should support them when other advanced stats (e.g. histograms) are available in spark. ## How was this patch tested? Add a new test suite. Author: Zhenhua Wang <wzh_zju@163.com> Author: wangzhenhua <wangzhenhua@huawei.com> Closes #16228 from wzhfy/joinEstimate.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 17 additions, 9 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala 18 additions, 2 deletions...alyst/plans/logical/statsEstimation/EstimationUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala 307 additions, 0 deletions...talyst/plans/logical/statsEstimation/JoinEstimation.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/Range.scala 116 additions, 0 deletions...rk/sql/catalyst/plans/logical/statsEstimation/Range.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala 327 additions, 0 deletions...rk/sql/catalyst/statsEstimation/JoinEstimationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/ProjectEstimationSuite.scala 9 additions, 12 deletions...sql/catalyst/statsEstimation/ProjectEstimationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/StatsEstimationTestBase.scala 7 additions, 1 deletion...ql/catalyst/statsEstimation/StatsEstimationTestBase.scala
Loading
Please register or sign in to comment