-
- Downloads
[SPARK-19514] Making range interruptible.
## What changes were proposed in this pull request? Previously range operator could not be interrupted. For example, using DAGScheduler.cancelStage(...) on a query with range might have been ineffective. This change adds periodic checks of TaskContext.isInterrupted to codegen version, and InterruptibleOperator to non-codegen version. I benchmarked the performance of codegen version on a sample query `spark.range(1000L * 1000 * 1000 * 10).count()` and there is no measurable difference. ## How was this patch tested? Adds a unit test. Author: Ala Luszczak <ala@databricks.com> Closes #16872 from ala/SPARK-19514b.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 5 additions, 3 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala 10 additions, 2 deletions...g/apache/spark/sql/execution/basicPhysicalOperators.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala 37 additions, 1 deletion...test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala
Loading
Please register or sign in to comment