-
- Downloads
[SPARK-12796] [SQL] Whole stage codegen
This is the initial work for whole stage codegen, it support Projection/Filter/Range, we will continue work on this to support more physical operators. A micro benchmark show that a query with range, filter and projection could be 3X faster then before. It's turned on by default. For a tree that have at least two chained plans, a WholeStageCodegen will be inserted into it, for example, the following plan ``` Limit 10 +- Project [(id#5L + 1) AS (id + 1)#6L] +- Filter ((id#5L & 1) = 1) +- Range 0, 1, 4, 10, [id#5L] ``` will be translated into ``` Limit 10 +- WholeStageCodegen +- Project [(id#1L + 1) AS (id + 1)#2L] +- Filter ((id#1L & 1) = 1) +- Range 0, 1, 4, 10, [id#1L] ``` Here is the call graph to generate Java source for A and B (A support codegen, but B does not): ``` * WholeStageCodegen Plan A FakeInput Plan B * ========================================================================= * * -> execute() * | * doExecute() --------> produce() * | * doProduce() -------> produce() * | * doProduce() ---> execute() * | * consume() * doConsume() ------------| * | * doConsume() <----- consume() ``` A SparkPlan that support codegen need to implement doProduce() and doConsume(): ``` def doProduce(ctx: CodegenContext): (RDD[InternalRow], String) def doConsume(ctx: CodegenContext, child: SparkPlan, input: Seq[ExprCode]): String ``` Author: Davies Liu <davies@databricks.com> Closes #10735 from davies/whole2.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala 5 additions, 1 deletion...pache/spark/sql/catalyst/expressions/BoundAttribute.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 42 additions, 34 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodegenFallback.scala 5 additions, 3 deletions...rk/sql/catalyst/expressions/codegen/CodegenFallback.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala 4 additions, 4 deletions...alyst/expressions/codegen/GenerateMutableProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala 4 additions, 4 deletions...k/sql/catalyst/expressions/codegen/GenerateOrdering.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala 4 additions, 4 deletions.../sql/catalyst/expressions/codegen/GeneratePredicate.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateSafeProjection.scala 4 additions, 4 deletions...catalyst/expressions/codegen/GenerateSafeProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala 4 additions, 6 deletions...talyst/expressions/codegen/GenerateUnsafeProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala 1 addition, 1 deletion...atalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala 1 addition, 0 deletions...ark/sql/catalyst/expressions/conditionalExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala 1 addition, 2 deletions...cala/org/apache/spark/sql/catalyst/expressions/misc.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 1 addition, 1 deletion.../scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 0 additions, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 9 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 2 additions, 1 deletion...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/BufferedRowIterator.java 64 additions, 0 deletions...a/org/apache/spark/sql/execution/BufferedRowIterator.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 0 additions, 1 deletion...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegen.scala 299 additions, 0 deletions...la/org/apache/spark/sql/execution/WholeStageCodegen.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 111 additions, 3 deletions...scala/org/apache/spark/sql/execution/basicOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala 3 additions, 3 deletions...spark/sql/execution/columnar/GenerateColumnAccessor.scala
Loading
Please register or sign in to comment