-
- Downloads
[SPARK-20229][SQL] add semanticHash to QueryPlan
## What changes were proposed in this pull request? Like `Expression`, `QueryPlan` should also have a `semanticHash` method, then we can put plans to a hash map and look it up fast. This PR refactors `QueryPlan` to follow `Expression` and put all the normalization logic in `QueryPlan.canonicalized`, so that it's very natural to implement `semanticHash`. follow-up: improve `CacheManager` to leverage this `semanticHash` and speed up plan lookup, instead of iterating all cached plans. ## How was this patch tested? existing tests. Note that we don't need to test the `semanticHash` method, once the existing tests prove `sameResult` is correct, we are good. Author: Wenchen Fan <wenchen@databricks.com> Closes #17541 from cloud-fan/plan-semantic.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 9 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 61 additions, 41 deletions...scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala 0 additions, 8 deletions...ache/spark/sql/catalyst/plans/logical/LocalRelation.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 0 additions, 2 deletions...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 2 additions, 0 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/broadcastMode.scala 2 additions, 7 deletions...che/spark/sql/catalyst/plans/physical/broadcastMode.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 17 additions, 20 deletions...a/org/apache/spark/sql/execution/DataSourceScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 0 additions, 14 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala 1 addition, 1 deletion...a/org/apache/spark/sql/execution/LocalTableScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala 3 additions, 7 deletions...g/apache/spark/sql/execution/basicPhysicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala 2 additions, 11 deletions...che/spark/sql/execution/datasources/LogicalRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala 2 additions, 4 deletions.../spark/sql/execution/exchange/BroadcastExchangeExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala 2 additions, 4 deletions...la/org/apache/spark/sql/execution/exchange/Exchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 3 additions, 8 deletions...org/apache/spark/sql/execution/joins/HashedRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeSuite.scala 9 additions, 9 deletions.../scala/org/apache/spark/sql/execution/ExchangeSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala 21 additions, 24 deletions...g/apache/spark/sql/hive/execution/HiveTableScanExec.scala
Loading
Please register or sign in to comment