-
- Downloads
[SPARK-9013][SQL] generate MutableProjection directly instead of return a function
`MutableProjection` is not thread-safe and we won't use it in multiple threads. I think the reason that we return `() => MutableProjection` is not about thread safety, but to save the costs of generating code when we need same but individual mutable projections. However, I only found one place that use this [feature](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala#L122-L123), and comparing to the troubles it brings, I think we should generate `MutableProjection` directly instead of return a function. Author: Wenchen Fan <wenchen@databricks.com> Closes #7373 from cloud-fan/project.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala 5 additions, 7 deletions...alyst/expressions/codegen/GenerateMutableProjection.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala 2 additions, 2 deletions.../spark/sql/catalyst/expressions/CodeGenerationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala 3 additions, 3 deletions...spark/sql/catalyst/expressions/ExpressionEvalHelper.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathFunctionsSuite.scala 1 addition, 1 deletion...e/spark/sql/catalyst/expressions/MathFunctionsSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodegenExpressionCachingSuite.scala 3 additions, 3 deletions...t/expressions/codegen/CodegenExpressionCachingSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala 1 addition, 1 deletion...talyst/expressions/codegen/GeneratedProjectionSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 1 addition, 3 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala 10 additions, 10 deletions...rc/main/scala/org/apache/spark/sql/execution/Window.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala 4 additions, 4 deletions...e/spark/sql/execution/aggregate/AggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala 1 addition, 1 deletion...ql/execution/aggregate/SortBasedAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala 1 addition, 1 deletion...che/spark/sql/execution/aggregate/TungstenAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 1 addition, 1 deletion...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/aggregate/udaf.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchPythonEvaluation.scala 1 addition, 1 deletion...he/spark/sql/execution/python/BatchPythonEvaluation.scala
Loading
Please register or sign in to comment