-
- Downloads
[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions
Conversion of outer joins, if the predicates in filter conditions can restrict the result sets so that all null-supplying rows are eliminated. - `full outer` -> `inner` if both sides have such predicates - `left outer` -> `inner` if the right side has such predicates - `right outer` -> `inner` if the left side has such predicates - `full outer` -> `left outer` if only the left side has such predicates - `full outer` -> `right outer` if only the right side has such predicates If applicable, this can greatly improve the performance, since outer join is much slower than inner join, full outer join is much slower than left/right outer join. The original PR is https://github.com/apache/spark/pull/10542 Author: gatorsmile <gatorsmile@gmail.com> Author: xiaoli <lixiao1983@gmail.com> Author: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local> Closes #10567 from gatorsmile/outerJoinEliminationByFilterCond.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 58 additions, 1 deletion...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala 195 additions, 0 deletions...rk/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala 48 additions, 0 deletions.../test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala 1 addition, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
Loading
Please register or sign in to comment