-
- Downloads
[SPARK-21273][SQL] Propagate logical plan stats using visitor pattern and mixin
## What changes were proposed in this pull request? We currently implement statistics propagation directly in logical plan. Given we already have two different implementations, it'd make sense to actually decouple the two and add stats propagation using mixin. This would reduce the coupling between logical plan and statistics handling. This can also be a powerful pattern in the future to add additional properties (e.g. constraints). ## How was this patch tested? Should be covered by existing test cases. Author: Reynold Xin <rxin@databricks.com> Closes #18479 from rxin/stats-trait.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala 2 additions, 3 deletions...ache/spark/sql/catalyst/plans/logical/LocalRelation.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 9 additions, 52 deletions...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlanVisitor.scala 87 additions, 0 deletions...spark/sql/catalyst/plans/logical/LogicalPlanVisitor.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 5 additions, 123 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala 0 additions, 5 deletions...a/org/apache/spark/sql/catalyst/plans/logical/hints.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/BasicStatsPlanVisitor.scala 82 additions, 0 deletions...plans/logical/statsEstimation/BasicStatsPlanVisitor.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/LogicalPlanStats.scala 50 additions, 0 deletions...lyst/plans/logical/statsEstimation/LogicalPlanStats.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala 163 additions, 0 deletions...cal/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/BasicStatsEstimationSuite.scala 0 additions, 44 deletions.../catalyst/statsEstimation/BasicStatsEstimationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/StatsEstimationTestBase.scala 1 addition, 1 deletion...ql/catalyst/statsEstimation/StatsEstimationTestBase.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 2 additions, 2 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala 1 addition, 1 deletion...pache/spark/sql/execution/columnar/InMemoryRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala 4 additions, 3 deletions...che/spark/sql/execution/datasources/LogicalRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala 1 addition, 2 deletions...ala/org/apache/spark/sql/execution/streaming/memory.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala 1 addition, 1 deletion...k/sql/hive/execution/PruneFileSourcePartitionsSuite.scala
Loading
Please register or sign in to comment